[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:38 dist-test-slave-wbpt krb5kdc[3814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:38 dist-test-slave-wbpt krb5kdc[3814](info): set up 2 sockets
Sep 29 17:47:38 dist-test-slave-wbpt krb5kdc[3814](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 17:47:39.740419 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:39 dist-test-slave-wbpt krb5kdc[3814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:39.859354 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41567
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:41567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:39.972704 3827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:39.972991 3827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:39.973049 3827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:39.977231 3827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:39.977304 3827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:39.977331 3827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:39.977353 3827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:39.977380 3827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:39.982978 3827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:41567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:39.984311 3827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:39.985481 3827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:39.995899 3833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:39.997427 3832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:39.997887 3835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:39.998199 3827 server_base.cc:1034] running on GCE node
I20260929 17:47:39.998852 3827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:40.000422 3827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:40.001591 3827 hybrid_clock.cc:648] HybridClock initialized: now 1790704060001568 us; error 35 us; skew 500 ppm
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:40.004443 3827 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:40.005509 3827 webserver.cc:460] Webserver started at http://127.3.183.254:38153/ using document root <none> and password file <none>
I20260929 17:47:40.006425 3827 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:40.006494 3827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:40.006752 3827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:40.009413 3827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "55e6f44e0a4d42cd86a80e9e6521be06"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "f2b3cfd647902b035d17d97a35a880b0"
server_key_iv: "b93e4df93f32cb0b2193c634d6bbc76f"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.010008 3827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "55e6f44e0a4d42cd86a80e9e6521be06"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "f2b3cfd647902b035d17d97a35a880b0"
server_key_iv: "b93e4df93f32cb0b2193c634d6bbc76f"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.014921 3827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 17:47:40.018602 3842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.020468 3827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:47:40.020581 3827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "55e6f44e0a4d42cd86a80e9e6521be06"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "f2b3cfd647902b035d17d97a35a880b0"
server_key_iv: "b93e4df93f32cb0b2193c634d6bbc76f"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.020695 3827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:40.068300 3827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:40.071798 3827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:40.071985 3827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:40.079181 3827 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:41567
I20260929 17:47:40.079254 3894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:41567 every 8 connection(s)
I20260929 17:47:40.080420 3827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:40.084311 3895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:40.089175 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 3827
I20260929 17:47:40.089350 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:40.091105 3807 external_mini_cluster.cc:1428] Setting key d899e5fc6dba0129773df3501f82aa9a
I20260929 17:47:40.092788 3895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06: Bootstrap starting.
I20260929 17:47:40.096565 3895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:40.097496 3895 log.cc:824] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:40.100301 3895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06: No bootstrap required, opened a new log
I20260929 17:47:40.105584 3895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41567 } }
I20260929 17:47:40.105849 3895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:40.105908 3895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e6f44e0a4d42cd86a80e9e6521be06, State: Initialized, Role: FOLLOWER
I20260929 17:47:40.106698 3895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41567 } }
I20260929 17:47:40.106896 3895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:40.106972 3895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:40.107071 3895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:40.108379 3895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41567 } }
I20260929 17:47:40.108906 3895 leader_election.cc:302] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e6f44e0a4d42cd86a80e9e6521be06; no voters:
I20260929 17:47:40.109292 3895 leader_election.cc:288] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:40.109459 3900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:40.109725 3900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [term 1 LEADER]: Becoming Leader. State: Replica: 55e6f44e0a4d42cd86a80e9e6521be06, State: Running, Role: LEADER
I20260929 17:47:40.110138 3900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41567 } }
I20260929 17:47:40.110981 3895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:40.112341 3901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41567 } } }
I20260929 17:47:40.112519 3901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:40.112341 3902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55e6f44e0a4d42cd86a80e9e6521be06. Latest consensus state: current_term: 1 leader_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e6f44e0a4d42cd86a80e9e6521be06" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41567 } } }
I20260929 17:47:40.112622 3902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06 [sys.catalog]: This master's current role is: LEADER
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
W20260929 17:47:40.116546 3915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:47:40.116631 3915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:47:40.116696 3910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:40.117678 3910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:40.121011 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:40.095603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.3.183.254:41567)
0929 17:47:40.096130 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:40.096150 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:40.097238 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 17:47:40.101153 (+ 3915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:40.101171 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:40.101487 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:40.101777 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:40.104821 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.106429 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.108455 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.109045 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.109200 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:40.114401 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:40.114456 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:40.114475 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:40.114510 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:40.116819 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.117528 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.117535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.117541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.117713 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.118200 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.118203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.118205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.118817 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:40.119020 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:40.119709 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 17:47:40.120032 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":169,"threads_started":1}
I20260929 17:47:40.124130 3910 catalog_manager.cc:1347] Generated new cluster ID: 26aa4d8adb434650a9e214dd25a9ded9
I20260929 17:47:40.124193 3910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:40.142023 3910 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:40.143368 3910 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:40.148375 3910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55e6f44e0a4d42cd86a80e9e6521be06: Generated new TSK 0
I20260929 17:47:40.149294 3910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:47:40.181419 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41567
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:40.333029 3921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:40.333285 3921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:40.333348 3921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:40.337580 3921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:40.337651 3921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:40.337731 3921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:47:40.343305 3921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:40.344410 3921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:40.345436 3921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:40.356555 3929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:40.356564 3926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:40.356595 3927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:40.356894 3921 server_base.cc:1034] running on GCE node
I20260929 17:47:40.357308 3921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:40.357975 3921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:40.359140 3921 hybrid_clock.cc:648] HybridClock initialized: now 1790704060359116 us; error 38 us; skew 500 ppm
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:40.362457 3921 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:47:40.363379 3921 webserver.cc:460] Webserver started at http://127.3.183.193:39421/ using document root <none> and password file <none>
I20260929 17:47:40.363966 3921 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:40.364033 3921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:40.364235 3921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:40.366391 3921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "b2fc4b0ccbc74d10a265fe34f5b46422"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "403f371647e2dd8e6701b4912ec500dc"
server_key_iv: "167eed58240413e395ad0b0006fd964a"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.366976 3921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "b2fc4b0ccbc74d10a265fe34f5b46422"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "403f371647e2dd8e6701b4912ec500dc"
server_key_iv: "167eed58240413e395ad0b0006fd964a"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.370999 3921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:47:40.373464 3936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.374758 3921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:47:40.374866 3921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "b2fc4b0ccbc74d10a265fe34f5b46422"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "403f371647e2dd8e6701b4912ec500dc"
server_key_iv: "167eed58240413e395ad0b0006fd964a"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.374961 3921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:40.406358 3921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:40.409852 3921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:40.410066 3921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:40.410771 3921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:40.411988 3921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:40.412087 3921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.412154 3921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:40.412179 3921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.429445 3921 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:39937
I20260929 17:47:40.429514 4049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:39937 every 8 connection(s)
I20260929 17:47:40.430334 3921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:47:40.440352 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 3921
I20260929 17:47:40.440438 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:47:40.440693 3807 external_mini_cluster.cc:1428] Setting key 6a151d3c6dc8f7a44d2b9ebb04ef2af6
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:40.444972 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:40.432503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:45037 (local address 127.3.183.254:41567)
0929 17:47:40.432633 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:40.432636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:40.433601 (+ 965us) server_negotiation.cc:407] Connection header received
0929 17:47:40.435183 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:40.435187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:40.435245 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:40.435313 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:40.436727 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.437281 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.438541 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.438772 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.438859 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:40.441649 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:40.441652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:40.441654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:40.441678 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:40.442982 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.443475 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.443477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.443479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.443561 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.443876 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.443879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.443881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.444119 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:40.444205 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:40.444693 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 17:47:40.444813 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:47:40.445981 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:40.432858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41567 (local address 127.3.183.193:45037)
0929 17:47:40.433445 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:40.433492 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:40.434974 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:40.435494 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:40.435501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:40.435998 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:40.436580 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:40.436595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.437394 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.437397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:40.438420 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:40.438431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.438912 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.438916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:40.439118 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:40.439714 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:40.439731 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:40.441494 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:40.443106 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:40.443112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:40.443128 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:40.443381 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:40.443647 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:40.443651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:40.443653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:40.443801 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:40.444219 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:40.444225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:40.444458 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 17:47:40.444620 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 17:47:40.444933 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":174,"threads_started":1}
I20260929 17:47:40.447162 4050 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41567
I20260929 17:47:40.447376 4050 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:40.447892 4050 heartbeater.cc:511] Master 127.3.183.254:41567 requested a full tablet report, sending...
I20260929 17:47:40.449471 3855 ts_manager.cc:194] Registered new tserver with Master: b2fc4b0ccbc74d10a265fe34f5b46422 (127.3.183.193:39937)
I20260929 17:47:40.450804 3855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:45037
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:47:40.480008 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41567
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:40.629302 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:40.629563 4055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:40.629626 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:40.633684 4055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:40.633744 4055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:40.633822 4055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:47:40.639315 4055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:40.640421 4055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:40.641405 4055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:40.651077 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:40.651150 4061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:40.651124 4060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:40.651340 4055 server_base.cc:1034] running on GCE node
I20260929 17:47:40.651859 4055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:40.652519 4055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:40.653672 4055 hybrid_clock.cc:648] HybridClock initialized: now 1790704060653645 us; error 40 us; skew 500 ppm
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:40.656865 4055 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:47:40.657764 4055 webserver.cc:460] Webserver started at http://127.3.183.194:46781/ using document root <none> and password file <none>
I20260929 17:47:40.658300 4055 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:40.658362 4055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:40.658550 4055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:40.660804 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "79d6b28bdbf6457a81c5138e679709d7"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "61b4dd5fc43759c110d159e23c22cee1"
server_key_iv: "dca72fb49c77e136eb8eeb4c11b522dd"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.661420 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "79d6b28bdbf6457a81c5138e679709d7"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "61b4dd5fc43759c110d159e23c22cee1"
server_key_iv: "dca72fb49c77e136eb8eeb4c11b522dd"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.665602 4055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:47:40.668076 4070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.669273 4055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:40.669376 4055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "79d6b28bdbf6457a81c5138e679709d7"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "61b4dd5fc43759c110d159e23c22cee1"
server_key_iv: "dca72fb49c77e136eb8eeb4c11b522dd"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.669467 4055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:40.684197 4055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:40.687486 4055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:40.687712 4055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:40.688484 4055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:40.689393 4055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:40.689442 4055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.689496 4055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:40.689522 4055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.706414 4055 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:33755
I20260929 17:47:40.706514 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:33755 every 8 connection(s)
I20260929 17:47:40.707415 4055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:47:40.717085 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4055
I20260929 17:47:40.717216 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:47:40.717500 3807 external_mini_cluster.cc:1428] Setting key 4b9ef775ee1d73eb3afb73c81608e4cb
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:40.722889 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:40.709435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:54843 (local address 127.3.183.254:41567)
0929 17:47:40.709607 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:40.709612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:40.710653 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 17:47:40.712547 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:40.712551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:40.712601 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:40.712678 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:40.714065 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.714592 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.715964 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.716196 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.716287 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:40.719295 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:40.719299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:40.719302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:40.719330 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:40.720792 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.721385 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.721387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.721389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.721464 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.721846 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.721848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.721850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.722017 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:40.722085 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:40.722611 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 17:47:40.722730 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 17:47:40.723853 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:40.709864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41567 (local address 127.3.183.194:54843)
0929 17:47:40.710499 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:40.710548 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:40.712314 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:40.712841 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:40.712851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:40.713306 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:40.713907 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:40.713934 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.714718 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.714721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:40.715822 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:40.715832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.716383 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.716386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:40.716615 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:40.717327 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:40.717345 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:40.719134 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:40.720931 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:40.720938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:40.720956 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:40.721295 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:40.721569 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:40.721572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:40.721574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:40.721758 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:40.722107 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:40.722113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:40.722373 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 17:47:40.722552 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 17:47:40.722883 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":178,"threads_started":1}
I20260929 17:47:40.724996 4184 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41567
I20260929 17:47:40.725224 4184 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:40.725704 4184 heartbeater.cc:511] Master 127.3.183.254:41567 requested a full tablet report, sending...
I20260929 17:47:40.726667 3855 ts_manager.cc:194] Registered new tserver with Master: 79d6b28bdbf6457a81c5138e679709d7 (127.3.183.194:33755)
I20260929 17:47:40.727226 3855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:54843
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:47:40.747354 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41567
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:40.897802 4189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:40.898106 4189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:40.898182 4189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:40.902315 4189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:40.902385 4189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:40.902479 4189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:47:40.908013 4189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:40.909106 4189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:40.910128 4189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:40.920899 4194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:40.920965 4195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:40.921047 4189 server_base.cc:1034] running on GCE node
W20260929 17:47:40.920981 4197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:40.921415 4189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:40.922024 4189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:40.923179 4189 hybrid_clock.cc:648] HybridClock initialized: now 1790704060923159 us; error 33 us; skew 500 ppm
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:40.926298 4189 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:47:40.927122 4189 webserver.cc:460] Webserver started at http://127.3.183.195:40159/ using document root <none> and password file <none>
I20260929 17:47:40.927644 4189 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:40.927706 4189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:40.927915 4189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:40.930054 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "a827b0e82363461dbce754ae8b3d1439"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "014342d23d6c8100c1d334683f603b69"
server_key_iv: "5d3ec0ee3ae858f429517f481bc80636"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.930655 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "a827b0e82363461dbce754ae8b3d1439"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "014342d23d6c8100c1d334683f603b69"
server_key_iv: "5d3ec0ee3ae858f429517f481bc80636"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.934738 4189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 17:47:40.937188 4204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.938504 4189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:47:40.938606 4189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "a827b0e82363461dbce754ae8b3d1439"
format_stamp: "Formatted at 2026-09-29 17:47:40 on dist-test-slave-wbpt"
server_key: "014342d23d6c8100c1d334683f603b69"
server_key_iv: "5d3ec0ee3ae858f429517f481bc80636"
server_key_version: "encryptionkey@0"
I20260929 17:47:40.938701 4189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:40.951287 4189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:40.954386 4189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:40.954587 4189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:40.955267 4189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:40.956176 4189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:40.956223 4189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.956274 4189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:40.956298 4189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:40.974241 4189 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:36837
I20260929 17:47:40.974368 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:36837 every 8 connection(s)
I20260929 17:47:40.975299 4189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:47:40.984616 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4189
I20260929 17:47:40.984736 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:47:40.985029 3807 external_mini_cluster.cc:1428] Setting key 2b6968f81746ab2aebf91e42154a1143
Sep 29 17:47:40 dist-test-slave-wbpt krb5kdc[3814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:40.990073 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:40.977281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:40539 (local address 127.3.183.254:41567)
0929 17:47:40.977407 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:40.977411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:40.978503 (+ 1092us) server_negotiation.cc:407] Connection header received
0929 17:47:40.980202 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:40.980205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:40.980260 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:40.980332 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:40.981626 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.982157 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.983382 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.983612 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.983700 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:40.986850 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:40.986852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:40.986855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:40.986878 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:40.988223 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.988682 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.988686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.988688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.988760 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:40.989051 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:40.989055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:40.989056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:40.989226 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:40.989307 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:40.989792 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 17:47:40.989905 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:47:40.990883 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:40.977695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41567 (local address 127.3.183.195:40539)
0929 17:47:40.978351 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:40.978402 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:40.980006 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:40.980507 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:40.980514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:40.980908 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:40.981487 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:40.981504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.982268 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.982271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:40.983270 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:40.983279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:40.983745 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:40.983749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:40.983974 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:40.984781 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:40.984801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:40.986697 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:40.988322 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:40.988327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:40.988344 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:40.988596 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:40.988838 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:40.988841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:40.988843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:40.988970 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:40.989316 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:40.989323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:40.989577 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 17:47:40.989737 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 17:47:40.990048 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":171,"threads_started":1}
I20260929 17:47:40.991989 4318 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41567
I20260929 17:47:40.992240 4318 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:40.992717 4318 heartbeater.cc:511] Master 127.3.183.254:41567 requested a full tablet report, sending...
I20260929 17:47:40.993703 3855 ts_manager.cc:194] Registered new tserver with Master: a827b0e82363461dbce754ae8b3d1439 (127.3.183.195:36837)
I20260929 17:47:40.994240 3855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:40539
I20260929 17:47:40.999116 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:47:41 dist-test-slave-wbpt krb5kdc[3814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:47:41 dist-test-slave-wbpt krb5kdc[3814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:47:41.026140 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:41.016400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.3.183.193:39937)
0929 17:47:41.016749 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:41.016756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:41.016865 (+ 109us) server_negotiation.cc:407] Connection header received
0929 17:47:41.016950 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:41.016957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:41.017164 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:41.017304 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:41.018107 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:41.018890 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:41.019808 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:41.020299 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:41.020384 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:41.022777 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:41.022786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:41.022800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:41.022825 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:41.024433 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:41.024852 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:41.024857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:41.024858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:41.024940 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:41.025181 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:41.025183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:41.025184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:41.025451 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:41.025619 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:41.025827 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 17:47:41.025952 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":129,"threads_started":1}
W20260929 17:47:41.027140 3964 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42794
Sep 29 17:47:41 dist-test-slave-wbpt krb5kdc[3814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:47:41 dist-test-slave-wbpt krb5kdc[3814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:47:41.051625 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:41.044014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.3.183.193:39937)
0929 17:47:41.044153 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:41.044156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:41.044190 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:47:41.044329 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:41.044332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:41.044373 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:41.044441 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:41.045063 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:41.045563 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:41.046365 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:41.046609 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:41.046680 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:41.048954 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:41.048958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:41.048960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:41.048983 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:41.050156 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:41.050552 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:41.050555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:41.050556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:41.050630 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:41.050860 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:41.050863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:41.050865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:41.051028 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:41.051138 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:41.051354 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 17:47:41.051464 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:47:41.052759 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 3921
I20260929 17:47:41.058382 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4055
I20260929 17:47:41.062908 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4189
I20260929 17:47:41.067368 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 3827
2026-09-29T17:47:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2292 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:41 dist-test-slave-wbpt krb5kdc[4332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:41 dist-test-slave-wbpt krb5kdc[4332](info): set up 2 sockets
Sep 29 17:47:41 dist-test-slave-wbpt krb5kdc[4332](info): commencing operation
krb5kdc: starting...
W20260929 17:47:42.027366 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:42.143491 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34547
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:42.250304 4345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:42.250556 4345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:42.250594 4345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:42.254563 4345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:42.254614 4345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:42.254633 4345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:42.254647 4345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:42.254663 4345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:42.259734 4345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:34547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:42.260716 4345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:42.261672 4345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:42.269719 4353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:42.269798 4350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:42.269851 4351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:42.269883 4345 server_base.cc:1034] running on GCE node
I20260929 17:47:42.270512 4345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:42.271553 4345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:42.272716 4345 hybrid_clock.cc:648] HybridClock initialized: now 1790704062272688 us; error 40 us; skew 500 ppm
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:42.275399 4345 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:42.276310 4345 webserver.cc:460] Webserver started at http://127.3.183.254:39501/ using document root <none> and password file <none>
I20260929 17:47:42.276818 4345 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:42.276883 4345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:42.277083 4345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:42.279140 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "66a558876e7c3a8b18b5e227d99d445c"
server_key_iv: "08330a24d9198c5af68e3103426a28c7"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.279731 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "66a558876e7c3a8b18b5e227d99d445c"
server_key_iv: "08330a24d9198c5af68e3103426a28c7"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.283368 4345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:47:42.285799 4360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.287092 4345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:42.287204 4345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "66a558876e7c3a8b18b5e227d99d445c"
server_key_iv: "08330a24d9198c5af68e3103426a28c7"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.287299 4345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:42.298187 4345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:42.300997 4345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:42.301154 4345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:42.307542 4345 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:34547
I20260929 17:47:42.307564 4412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:34547 every 8 connection(s)
I20260929 17:47:42.308405 4345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:42.308917 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4345
I20260929 17:47:42.309024 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:42.309247 3807 external_mini_cluster.cc:1428] Setting key 4c8f72ad445610a1329fc80df3b76e76
I20260929 17:47:42.311349 4413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:42.317975 4413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b: Bootstrap starting.
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:42.320658 4413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:42.321447 4413 log.cc:824] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:42.323369 4413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b: No bootstrap required, opened a new log
I20260929 17:47:42.325136 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:42.310637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.3.183.254:34547)
0929 17:47:42.311042 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:42.311061 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:42.311102 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:47:42.311938 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:42.311960 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:42.312347 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:42.312704 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:42.313667 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.315484 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.316736 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.317416 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.317577 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:42.320551 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:42.320557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:42.320573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:42.320605 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:42.322204 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:42.322667 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:42.322673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:42.322677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:42.322805 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:42.323080 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:42.323082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:42.323084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:42.323536 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:42.323696 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:42.323970 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 17:47:42.324198 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20260929 17:47:42.326403 4413 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34547 } }
I20260929 17:47:42.326612 4413 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:42.326668 4413 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a5eea18c8d44c7a4fd18c2cbdbab6b, State: Initialized, Role: FOLLOWER
I20260929 17:47:42.327078 4413 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34547 } }
I20260929 17:47:42.327221 4413 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:42.327287 4413 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:42.327380 4413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:42.328248 4413 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34547 } }
I20260929 17:47:42.328559 4413 leader_election.cc:302] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a5eea18c8d44c7a4fd18c2cbdbab6b; no voters:
I20260929 17:47:42.328868 4413 leader_election.cc:288] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:42.328950 4418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:42.329145 4418 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [term 1 LEADER]: Becoming Leader. State: Replica: d6a5eea18c8d44c7a4fd18c2cbdbab6b, State: Running, Role: LEADER
I20260929 17:47:42.329418 4418 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34547 } }
I20260929 17:47:42.330242 4413 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:42.331264 4419 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34547 } } }
I20260929 17:47:42.331404 4419 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:42.331283 4420 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a5eea18c8d44c7a4fd18c2cbdbab6b. Latest consensus state: current_term: 1 leader_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a5eea18c8d44c7a4fd18c2cbdbab6b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34547 } } }
I20260929 17:47:42.331676 4420 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:42.331817 4427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:42.334589 4427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:42.339654 4427 catalog_manager.cc:1347] Generated new cluster ID: cf6b75efa05440f8a19e4afdb676d95a
I20260929 17:47:42.339721 4427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:42.349220 4427 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:42.350447 4427 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:42.360987 4427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a5eea18c8d44c7a4fd18c2cbdbab6b: Generated new TSK 0
I20260929 17:47:42.361876 4427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:47:42.390395 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34547
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:42.501019 4439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:42.501269 4439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:42.501332 4439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:42.505367 4439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:42.505422 4439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:42.505505 4439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:47:42.510582 4439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34547
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:42.511567 4439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:42.512611 4439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:42.522428 4445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:42.522406 4444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:42.522612 4439 server_base.cc:1034] running on GCE node
W20260929 17:47:42.522408 4447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:42.523236 4439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:42.523900 4439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:42.525080 4439 hybrid_clock.cc:648] HybridClock initialized: now 1790704062525058 us; error 35 us; skew 500 ppm
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:42.527875 4439 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:47:42.528751 4439 webserver.cc:460] Webserver started at http://127.3.183.193:33315/ using document root <none> and password file <none>
I20260929 17:47:42.529261 4439 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:42.529317 4439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:42.529476 4439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:42.531548 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "5a60bea45163430d996dcce99477bca2"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "ff35394e3473867bb9d744db0c809e64"
server_key_iv: "4807aa744f553078d546e18bd43b4991"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.532114 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "5a60bea45163430d996dcce99477bca2"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "ff35394e3473867bb9d744db0c809e64"
server_key_iv: "4807aa744f553078d546e18bd43b4991"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.535863 4439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:47:42.538301 4454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.539564 4439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:42.539671 4439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "5a60bea45163430d996dcce99477bca2"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "ff35394e3473867bb9d744db0c809e64"
server_key_iv: "4807aa744f553078d546e18bd43b4991"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.539839 4439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:42.549371 4439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:42.552373 4439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:42.552554 4439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:42.553215 4439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:42.554033 4439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:42.554080 4439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.554121 4439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:42.554144 4439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.565059 4439 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:39887
I20260929 17:47:42.565099 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:39887 every 8 connection(s)
I20260929 17:47:42.565973 4439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:47:42.566543 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4439
I20260929 17:47:42.566665 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:47:42.566998 3807 external_mini_cluster.cc:1428] Setting key d51f13641e59ac5193fd6ef126aab44e
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:47:42.580497 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:42.568082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:54835 (local address 127.3.183.254:34547)
0929 17:47:42.568220 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:42.568224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:42.569313 (+ 1089us) server_negotiation.cc:407] Connection header received
0929 17:47:42.571017 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:42.571020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:42.571066 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:42.571140 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:42.572548 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.573095 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.574265 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.574500 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.574590 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:42.577133 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:42.577136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:42.577139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:42.577188 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:42.578429 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:42.578895 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:42.578898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:42.578900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:42.579010 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:42.579356 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:42.579359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:42.579360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:42.579546 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:42.579637 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:42.580257 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 17:47:42.580371 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:47:42.581543 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:42.568549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34547 (local address 127.3.183.193:54835)
0929 17:47:42.569135 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:42.569196 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:42.570768 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:42.571360 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:42.571370 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:42.571880 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:42.572389 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:42.572402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.573244 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.573247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:42.574135 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:42.574144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.574645 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.574649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:42.574819 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:42.575377 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:42.575393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:42.576946 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:42.578535 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:42.578545 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:42.578557 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:42.578804 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:42.579119 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:42.579122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:42.579124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:42.579265 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:42.579667 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:42.579674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:42.579994 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 17:47:42.580208 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 17:47:42.580558 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":153,"threads_started":1}
I20260929 17:47:42.582762 4568 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34547
I20260929 17:47:42.583030 4568 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:42.583669 4568 heartbeater.cc:511] Master 127.3.183.254:34547 requested a full tablet report, sending...
I20260929 17:47:42.585312 4377 ts_manager.cc:194] Registered new tserver with Master: 5a60bea45163430d996dcce99477bca2 (127.3.183.193:39887)
I20260929 17:47:42.586418 4377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:54835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:47:42.594182 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34547
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:42.707801 4573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:42.708094 4573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:42.708143 4573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:42.712199 4573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:42.712263 4573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:42.712329 4573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:47:42.717622 4573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34547
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:42.718835 4573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:42.719874 4573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:42.729183 4578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:42.729189 4579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:42.729207 4573 server_base.cc:1034] running on GCE node
W20260929 17:47:42.729189 4581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:42.729923 4573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:42.730574 4573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:42.731737 4573 hybrid_clock.cc:648] HybridClock initialized: now 1790704062731715 us; error 34 us; skew 500 ppm
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:42.734323 4573 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:47:42.735123 4573 webserver.cc:460] Webserver started at http://127.3.183.194:35687/ using document root <none> and password file <none>
I20260929 17:47:42.735637 4573 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:42.735698 4573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:42.735980 4573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:42.738029 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "7712d179b865d520eb22ab6a2d33cacf"
server_key_iv: "71f9aaf3eff536cd958fbf62fa8a5027"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.738529 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "7712d179b865d520eb22ab6a2d33cacf"
server_key_iv: "71f9aaf3eff536cd958fbf62fa8a5027"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.742147 4573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 17:47:42.744668 4588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.745847 4573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 17:47:42.745944 4573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "7712d179b865d520eb22ab6a2d33cacf"
server_key_iv: "71f9aaf3eff536cd958fbf62fa8a5027"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.746047 4573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:42.773977 4573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:42.777343 4573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:42.777514 4573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:42.778120 4573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:42.779055 4573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:42.779110 4573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.779150 4573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:42.779174 4573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.788178 4573 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:38131
I20260929 17:47:42.788195 4701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:38131 every 8 connection(s)
I20260929 17:47:42.789134 4573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:47:42.790263 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4573
I20260929 17:47:42.790361 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:47:42.790669 3807 external_mini_cluster.cc:1428] Setting key 5d38fb53924fff0ac10881400719e0e5
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:47:42.803648 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:42.791125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:44493 (local address 127.3.183.254:34547)
0929 17:47:42.791288 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:42.791293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:42.792475 (+ 1182us) server_negotiation.cc:407] Connection header received
0929 17:47:42.794182 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:42.794186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:42.794240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:42.794326 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:42.795523 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.796339 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.797511 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.797847 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.797954 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:42.800257 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:42.800260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:42.800262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:42.800286 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:42.801613 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:42.802135 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:42.802139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:42.802141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:42.802215 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:42.802581 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:42.802584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:42.802586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:42.802758 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:42.802840 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:42.803368 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 17:47:42.803486 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:47:42.804512 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:42.791693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34547 (local address 127.3.183.194:44493)
0929 17:47:42.792311 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:42.792370 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:42.793909 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:42.794494 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:42.794501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:42.794865 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:42.795380 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:42.795393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.796462 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.796465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:42.797387 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:42.797396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:42.797998 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:42.798000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:42.798141 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:42.798731 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:42.798746 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:42.800110 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:42.801712 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:42.801717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:42.801729 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:42.802047 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:42.802302 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:42.802305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:42.802307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:42.802442 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:42.802852 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:42.802859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:42.803141 (+ 282us) client_negotiation.cc:769] Sending connection context
0929 17:47:42.803310 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:47:42.803640 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":161,"threads_started":1}
I20260929 17:47:42.805724 4702 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34547
I20260929 17:47:42.805948 4702 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:42.806420 4702 heartbeater.cc:511] Master 127.3.183.254:34547 requested a full tablet report, sending...
I20260929 17:47:42.807453 4377 ts_manager.cc:194] Registered new tserver with Master: b4963bb0261c4e3f8e54a2c2fd2ee827 (127.3.183.194:38131)
I20260929 17:47:42.808036 4377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:44493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:47:42.818984 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34547
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:42.926432 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:42.926690 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:42.926750 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:42.930792 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:42.930855 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:42.930933 4707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:47:42.936327 4707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34547
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:42.937402 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:42.938433 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:42.947880 4713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:42.947845 4712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:42.947995 4707 server_base.cc:1034] running on GCE node
W20260929 17:47:42.948140 4715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:42.948583 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:42.949246 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:42.950423 4707 hybrid_clock.cc:648] HybridClock initialized: now 1790704062950394 us; error 38 us; skew 500 ppm
Sep 29 17:47:42 dist-test-slave-wbpt krb5kdc[4332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:42.958098 4707 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:47:42.959079 4707 webserver.cc:460] Webserver started at http://127.3.183.195:37377/ using document root <none> and password file <none>
I20260929 17:47:42.959604 4707 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:42.959661 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:42.959872 4707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:42.961872 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "c5832ddb6d68ce5e6849c25d47976d57"
server_key_iv: "95ea6b02e95dd13aba4edc95f1f30da9"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.962446 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "c5832ddb6d68ce5e6849c25d47976d57"
server_key_iv: "95ea6b02e95dd13aba4edc95f1f30da9"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.966923 4707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:47:42.969842 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.971256 4707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:47:42.971364 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6"
format_stamp: "Formatted at 2026-09-29 17:47:42 on dist-test-slave-wbpt"
server_key: "c5832ddb6d68ce5e6849c25d47976d57"
server_key_iv: "95ea6b02e95dd13aba4edc95f1f30da9"
server_key_version: "encryptionkey@0"
I20260929 17:47:42.971489 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:42.981441 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:42.984647 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:42.984827 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:42.985397 4707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:42.986186 4707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:42.986230 4707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.986281 4707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:42.986310 4707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:42.995239 4707 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:39097
I20260929 17:47:42.995276 4835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:39097 every 8 connection(s)
I20260929 17:47:42.996290 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:47:43.005249 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4707
I20260929 17:47:43.005373 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:47:43.005684 3807 external_mini_cluster.cc:1428] Setting key efa907f14742e4744263e8776dbd477d
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:43.010094 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:42.998222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:53415 (local address 127.3.183.254:34547)
0929 17:47:42.998340 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:42.998344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:42.999287 (+ 943us) server_negotiation.cc:407] Connection header received
0929 17:47:43.000694 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.000698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.000759 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.000832 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:43.002091 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.002622 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.003756 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.004010 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.004103 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.006638 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:43.006643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:43.006645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:43.006683 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:43.008190 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.008668 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.008671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.008672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.008746 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.009055 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.009058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.009059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.009238 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:43.009324 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.009808 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.009928 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:47:43.011020 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:42.998621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34547 (local address 127.3.183.195:53415)
0929 17:47:42.999107 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:42.999154 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:43.000463 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:43.001022 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:43.001029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:43.001401 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:43.001894 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.001908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.002750 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.002752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.003633 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.003643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.004150 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.004154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.004323 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.004854 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:43.004870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:43.006452 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:43.008297 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:43.008303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:43.008316 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:43.008573 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:43.008835 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:43.008837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:43.008838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:43.008966 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:43.009332 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:43.009339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:43.009585 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 17:47:43.009752 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 17:47:43.010070 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":133,"threads_started":1}
I20260929 17:47:43.012204 4836 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34547
I20260929 17:47:43.012458 4836 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:43.012938 4836 heartbeater.cc:511] Master 127.3.183.254:34547 requested a full tablet report, sending...
I20260929 17:47:43.013926 4377 ts_manager.cc:194] Registered new tserver with Master: f6e26c85ce3444c8a7d5ea77b54fd6f6 (127.3.183.195:39097)
I20260929 17:47:43.014493 4377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:53415
I20260929 17:47:43.019402 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:47:43.030624 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.023235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.3.183.254:34547)
0929 17:47:43.023381 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.023385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.023466 (+ 81us) server_negotiation.cc:407] Connection header received
0929 17:47:43.023605 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.023609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.023651 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.023720 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:43.024446 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.024989 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.025794 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.026054 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.026144 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.027560 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:43.027563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:43.027565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:43.027587 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:43.028999 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.029380 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.029384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.029386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.029467 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.029700 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.029703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.029705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.029969 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:43.030097 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.030307 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.030416 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 17:47:43.036548 4377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:47:43.039914 4377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:43.054785 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.049884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.3.183.194:38131)
0929 17:47:43.050234 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.050246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.050264 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:47:43.050323 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.050327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.050489 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.050939 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:43.051098 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.051824 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.053370 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.054107 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.054202 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.054437 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.054531 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.054623 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":157,"threads_started":1}
I20260929 17:47:43.054782 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.049193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:38131 (local address 127.0.0.1:54684)
0929 17:47:43.049615 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:43.049650 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:43.049787 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:43.050654 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:43.050660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:43.050683 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:43.050799 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.050805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.051933 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.051937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.053247 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.053257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.054223 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.054227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.054350 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.054392 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 17:47:43.054494 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 17:47:43.054623 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":29,"spinlock_wait_cycles":11264,"thread_start_us":92,"threads_started":1}
I20260929 17:47:43.055176 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.048956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39887 (local address 127.0.0.1:38800)
0929 17:47:43.049930 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:43.049938 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:43.050018 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:43.050702 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:43.050705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:43.050721 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:43.050832 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.050838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.052354 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.052357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.053540 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.053549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.054768 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.054771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.054859 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.054881 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:47:43.054945 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:47:43.055013 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":18,"thread_start_us":103,"threads_started":1}
I20260929 17:47:43.055480 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.049169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.3.183.195:39097)
0929 17:47:43.050414 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.050425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.050447 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:47:43.050520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.050526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.050728 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.050883 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:43.051091 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.052230 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.053839 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.054593 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.054680 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.054788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.055240 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.055339 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":125,"threads_started":1}
I20260929 17:47:43.055734 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.049545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.3.183.193:39887)
0929 17:47:43.049823 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.049837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.050101 (+ 264us) server_negotiation.cc:407] Connection header received
0929 17:47:43.050183 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.050188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.050397 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.050546 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:43.050955 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.052219 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.053667 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.054455 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.054570 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.054707 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.055461 (+ 754us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.055597 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260929 17:47:43.055176 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.048817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39097 (local address 127.0.0.1:34250)
0929 17:47:43.049616 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:43.049652 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:43.049787 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:43.050870 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:43.050873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:43.050890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:43.050968 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.050973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.052390 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.052392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.053713 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.053724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.054838 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.054841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.054922 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.054941 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:47:43.054993 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 17:47:43.055060 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":211,"spinlock_wait_cycles":8448,"thread_start_us":84,"threads_started":1}
I20260929 17:47:43.057986 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 587199fa86ad4427a697e4f8b97c7c34 (DEFAULT_TABLE table=test-table [id=a5f1a0a8e2904e4dabbff66f3d3b4f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:43.058225 4502 tablet_service.cc:1467] Processing CreateTablet for tablet 587199fa86ad4427a697e4f8b97c7c34 (DEFAULT_TABLE table=test-table [id=a5f1a0a8e2904e4dabbff66f3d3b4f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:43.059051 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587199fa86ad4427a697e4f8b97c7c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:43.059083 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587199fa86ad4427a697e4f8b97c7c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:43.061797 4770 tablet_service.cc:1467] Processing CreateTablet for tablet 587199fa86ad4427a697e4f8b97c7c34 (DEFAULT_TABLE table=test-table [id=a5f1a0a8e2904e4dabbff66f3d3b4f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:43.062811 4770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587199fa86ad4427a697e4f8b97c7c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:43.065021 4853 tablet_bootstrap.cc:492] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827: Bootstrap starting.
I20260929 17:47:43.066332 4854 tablet_bootstrap.cc:492] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2: Bootstrap starting.
I20260929 17:47:43.067638 4853 tablet_bootstrap.cc:654] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:43.068367 4853 log.cc:824] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:43.068758 4854 tablet_bootstrap.cc:654] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:43.069933 4854 log.cc:824] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:43.070075 4856 tablet_bootstrap.cc:492] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6: Bootstrap starting.
I20260929 17:47:43.070200 4853 tablet_bootstrap.cc:492] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827: No bootstrap required, opened a new log
I20260929 17:47:43.070408 4853 ts_tablet_manager.cc:1397] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 17:47:43.071619 4854 tablet_bootstrap.cc:492] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2: No bootstrap required, opened a new log
I20260929 17:47:43.071810 4854 ts_tablet_manager.cc:1397] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:47:43.072789 4856 tablet_bootstrap.cc:654] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:43.073820 4856 log.cc:824] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:43.075191 4853 raft_consensus.cc:348] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.075532 4853 raft_consensus.cc:374] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:43.075615 4853 raft_consensus.cc:729] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4963bb0261c4e3f8e54a2c2fd2ee827, State: Initialized, Role: FOLLOWER
I20260929 17:47:43.075816 4856 tablet_bootstrap.cc:492] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6: No bootstrap required, opened a new log
I20260929 17:47:43.075687 4854 raft_consensus.cc:348] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.075953 4854 raft_consensus.cc:374] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:43.076030 4856 ts_tablet_manager.cc:1397] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:47:43.076086 4853 consensus_queue.cc:260] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.076035 4854 raft_consensus.cc:729] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60bea45163430d996dcce99477bca2, State: Initialized, Role: FOLLOWER
I20260929 17:47:43.076578 4854 consensus_queue.cc:260] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.076879 4702 heartbeater.cc:503] Master 127.3.183.254:34547 was elected leader, sending a full tablet report...
I20260929 17:47:43.077229 4853 ts_tablet_manager.cc:1428] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260929 17:47:43.078186 4854 ts_tablet_manager.cc:1428] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:47:43.078523 4568 heartbeater.cc:503] Master 127.3.183.254:34547 was elected leader, sending a full tablet report...
I20260929 17:47:43.080904 4856 raft_consensus.cc:348] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.081158 4856 raft_consensus.cc:374] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:43.081259 4856 raft_consensus.cc:729] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e26c85ce3444c8a7d5ea77b54fd6f6, State: Initialized, Role: FOLLOWER
I20260929 17:47:43.081615 4856 consensus_queue.cc:260] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.082391 4856 ts_tablet_manager.cc:1428] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:47:43.082460 4836 heartbeater.cc:503] Master 127.3.183.254:34547 was elected leader, sending a full tablet report...
I20260929 17:47:43.235538 4859 raft_consensus.cc:482] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:43.235872 4859 raft_consensus.cc:504] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.237690 4859 leader_election.cc:288] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6e26c85ce3444c8a7d5ea77b54fd6f6 (127.3.183.195:39097), 5a60bea45163430d996dcce99477bca2 (127.3.183.193:39887)
I20260929 17:47:43.241554 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.237966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39097 (local address 127.3.183.194:57301)
0929 17:47:43.238074 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:43.238089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:43.238186 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:43.238633 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:43.238637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:43.238654 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:43.238837 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.238844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.239577 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.239581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.240444 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.240453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.241123 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.241127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.241269 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.241288 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:47:43.241353 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:47:43.241425 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 17:47:43.241554 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.238294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:57301 (local address 127.3.183.195:39097)
0929 17:47:43.238434 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.238437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.238449 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:47:43.238493 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.238495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.238532 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.238597 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:43.238958 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.239475 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.240565 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.241016 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.241123 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.241187 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.241380 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.241453 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:47:43.241950 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.238039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:35461 (local address 127.3.183.193:39887)
0929 17:47:43.238172 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.238177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.238611 (+ 434us) server_negotiation.cc:407] Connection header received
0929 17:47:43.238717 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.238720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.238767 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.238825 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:43.239171 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.239656 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.240986 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.241373 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.241479 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.241526 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.241761 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.241836 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:47:43.242336 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.238001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39887 (local address 127.3.183.194:35461)
0929 17:47:43.238439 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:43.238450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:43.238631 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:43.238839 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:43.238842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:43.238856 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:43.239058 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.239065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.239750 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.239753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.240871 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:43.240880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.241473 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.241476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:43.241616 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.241640 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:47:43.242146 (+ 506us) client_negotiation.cc:240] Negotiation successful
0929 17:47:43.242207 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":168,"threads_started":1}
I20260929 17:47:43.242319 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587199fa86ad4427a697e4f8b97c7c34" candidate_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" is_pre_election: true
I20260929 17:47:43.242682 4790 raft_consensus.cc:2454] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4963bb0261c4e3f8e54a2c2fd2ee827 in term 0.
I20260929 17:47:43.242956 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587199fa86ad4427a697e4f8b97c7c34" candidate_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60bea45163430d996dcce99477bca2" is_pre_election: true
I20260929 17:47:43.243217 4522 raft_consensus.cc:2454] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4963bb0261c4e3f8e54a2c2fd2ee827 in term 0.
I20260929 17:47:43.243252 4591 leader_election.cc:302] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4963bb0261c4e3f8e54a2c2fd2ee827, f6e26c85ce3444c8a7d5ea77b54fd6f6; no voters:
I20260929 17:47:43.243522 4859 raft_consensus.cc:2781] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:43.243604 4859 raft_consensus.cc:482] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:43.243655 4859 raft_consensus.cc:3037] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:43.245093 4859 raft_consensus.cc:504] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.245417 4859 leader_election.cc:288] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [CANDIDATE]: Term 1 election: Requested vote from peers f6e26c85ce3444c8a7d5ea77b54fd6f6 (127.3.183.195:39097), 5a60bea45163430d996dcce99477bca2 (127.3.183.193:39887)
I20260929 17:47:43.245743 4522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587199fa86ad4427a697e4f8b97c7c34" candidate_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60bea45163430d996dcce99477bca2"
I20260929 17:47:43.245776 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587199fa86ad4427a697e4f8b97c7c34" candidate_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6"
I20260929 17:47:43.245873 4790 raft_consensus.cc:3037] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:43.245872 4522 raft_consensus.cc:3037] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2 [term 0 FOLLOWER]: Advancing to term 1
W20260929 17:47:43.247375 4837 tablet.cc:2369] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:47:43.247498 4522 raft_consensus.cc:2454] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4963bb0261c4e3f8e54a2c2fd2ee827 in term 1.
I20260929 17:47:43.247721 4790 raft_consensus.cc:2454] T 587199fa86ad4427a697e4f8b97c7c34 P f6e26c85ce3444c8a7d5ea77b54fd6f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4963bb0261c4e3f8e54a2c2fd2ee827 in term 1.
I20260929 17:47:43.248098 4590 leader_election.cc:302] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5a60bea45163430d996dcce99477bca2, b4963bb0261c4e3f8e54a2c2fd2ee827, f6e26c85ce3444c8a7d5ea77b54fd6f6; no voters:
I20260929 17:47:43.248284 4859 raft_consensus.cc:2781] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:43.248453 4859 raft_consensus.cc:686] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [term 1 LEADER]: Becoming Leader. State: Replica: b4963bb0261c4e3f8e54a2c2fd2ee827, State: Running, Role: LEADER
I20260929 17:47:43.248778 4859 consensus_queue.cc:237] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } }
I20260929 17:47:43.251912 4377 catalog_manager.cc:5495] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4963bb0261c4e3f8e54a2c2fd2ee827 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e26c85ce3444c8a7d5ea77b54fd6f6" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a60bea45163430d996dcce99477bca2" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4963bb0261c4e3f8e54a2c2fd2ee827" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38131 } health_report { overall_health: HEALTHY } } }
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
W20260929 17:47:43.290222 4703 tablet.cc:2369] T 587199fa86ad4427a697e4f8b97c7c34 P b4963bb0261c4e3f8e54a2c2fd2ee827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:47:43.294144 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.284480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.3.183.193:39887)
0929 17:47:43.284637 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.284641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.284995 (+ 354us) server_negotiation.cc:407] Connection header received
0929 17:47:43.285126 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.285130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.285176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.285242 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:43.286044 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.286548 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.287627 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.287894 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.287992 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.290782 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:43.290794 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:43.290809 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:43.290834 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:43.292668 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.293051 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.293058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.293059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.293172 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.293364 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.293366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.293367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.293594 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:43.293740 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.293873 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.293961 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704063, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 17:47:43.317503 4569 tablet.cc:2369] T 587199fa86ad4427a697e4f8b97c7c34 P 5a60bea45163430d996dcce99477bca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704063, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:47:43.323379 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.314750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.3.183.193:39887)
0929 17:47:43.314867 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.314871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.315019 (+ 148us) server_negotiation.cc:407] Connection header received
0929 17:47:43.315149 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.315153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.315197 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.315261 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:43.315991 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.316550 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.317371 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.317660 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.317758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.320413 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:43.320417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:43.320420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:43.320451 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:43.322026 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.322386 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.322388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.322389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.322495 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.322678 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.322680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.322681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.322835 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:43.322927 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.323138 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.323232 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:47:43.348170 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:43.340349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.3.183.193:39887)
0929 17:47:43.340477 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:43.340481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:43.340642 (+ 161us) server_negotiation.cc:407] Connection header received
0929 17:47:43.340772 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:43.340775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:43.340818 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:43.340885 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:43.341545 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.342042 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.342915 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:43.343137 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:43.343221 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:43.345466 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:43.345470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:43.345472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:43.345495 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:43.346657 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.347038 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.347042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.347043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.347117 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:43.347344 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:43.347347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:43.347349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:43.347507 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:43.347615 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:43.347884 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 17:47:43.348004 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 17:47:43.349547 4482 tablet_service.cc:2908] Rejecting scan request for tablet 587199fa86ad4427a697e4f8b97c7c34: Uninitialized: clean time has not yet been initialized
I20260929 17:47:43.351686 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4439
I20260929 17:47:43.356372 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4573
I20260929 17:47:43.360950 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4707
I20260929 17:47:43.365201 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4345
2026-09-29T17:47:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2297 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4877](info): set up 2 sockets
Sep 29 17:47:43 dist-test-slave-wbpt krb5kdc[4877](info): commencing operation
krb5kdc: starting...
W20260929 17:47:44.311684 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:44 dist-test-slave-wbpt krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:44.426076 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38313
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:38313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:44.531126 4890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:44.531383 4890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:44.531421 4890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:44.535425 4890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:44.535502 4890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:47:44.535519 4890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:44.535537 4890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:44.535550 4890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:44.535567 4890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:44.540964 4890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:38313
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:44.542061 4890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:44.543094 4890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:44.551332 4895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:44.551345 4898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:44.551340 4896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:44.551597 4890 server_base.cc:1034] running on GCE node
I20260929 17:47:44.552219 4890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:44.553318 4890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:44.554489 4890 hybrid_clock.cc:648] HybridClock initialized: now 1790704064554468 us; error 35 us; skew 500 ppm
Sep 29 17:47:44 dist-test-slave-wbpt krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704064, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:44.557184 4890 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:44.558029 4890 webserver.cc:460] Webserver started at http://127.3.183.254:39067/ using document root <none> and password file <none>
I20260929 17:47:44.558590 4890 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:44.558660 4890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:44.558866 4890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:44.560873 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "31fbb737c0a240cda15e2a43c1f18ba6"
format_stamp: "Formatted at 2026-09-29 17:47:44 on dist-test-slave-wbpt"
server_key: "5666c2f92790b30cd20dd52b2495e258"
server_key_iv: "5bc7b9e54fb382f90da1838400c49b53"
server_key_version: "encryptionkey@0"
I20260929 17:47:44.561401 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "31fbb737c0a240cda15e2a43c1f18ba6"
format_stamp: "Formatted at 2026-09-29 17:47:44 on dist-test-slave-wbpt"
server_key: "5666c2f92790b30cd20dd52b2495e258"
server_key_iv: "5bc7b9e54fb382f90da1838400c49b53"
server_key_version: "encryptionkey@0"
I20260929 17:47:44.564883 4890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 17:47:44.567095 4905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:44.568316 4890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:44.568423 4890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "31fbb737c0a240cda15e2a43c1f18ba6"
format_stamp: "Formatted at 2026-09-29 17:47:44 on dist-test-slave-wbpt"
server_key: "5666c2f92790b30cd20dd52b2495e258"
server_key_iv: "5bc7b9e54fb382f90da1838400c49b53"
server_key_version: "encryptionkey@0"
I20260929 17:47:44.568519 4890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:44.584664 4890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:44.587508 4890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:44.587687 4890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:44.595618 4890 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:38313
I20260929 17:47:44.595705 4967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:38313 every 8 connection(s)
I20260929 17:47:44.596606 4890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:44.599373 4968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:44.601645 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4890
I20260929 17:47:44.601763 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:44.602034 3807 external_mini_cluster.cc:1428] Setting key 7c4ce8d30dba9926f827ff010ebfc872
I20260929 17:47:44.605197 4968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: Bootstrap starting.
I20260929 17:47:44.607652 4968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:44.608501 4968 log.cc:824] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:44.610679 4968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: No bootstrap required, opened a new log
Sep 29 17:47:44 dist-test-slave-wbpt krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:44.614058 4968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38313 } }
I20260929 17:47:44.614286 4968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:44.614343 4968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31fbb737c0a240cda15e2a43c1f18ba6, State: Initialized, Role: FOLLOWER
I20260929 17:47:44.614786 4968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38313 } }
I20260929 17:47:44.614915 4968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:44.614976 4968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:44.615074 4968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:44.615959 4968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38313 } }
I20260929 17:47:44.616194 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:44.603459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.3.183.254:38313)
0929 17:47:44.603896 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:44.603912 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:44.603966 (+ 54us) server_negotiation.cc:407] Connection header received
0929 17:47:44.604597 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:44.604612 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:44.604955 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:44.605240 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:44.606095 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.607159 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.608310 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.608848 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.609045 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:44.611497 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:44.611504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:44.611520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:44.611545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:44.613194 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:44.613684 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:44.613689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:44.613694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:44.613814 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:44.614076 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:44.614080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:44.614081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:44.614514 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:44.614711 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:44.614953 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 17:47:44.615383 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260929 17:47:44.616269 4968 leader_election.cc:302] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31fbb737c0a240cda15e2a43c1f18ba6; no voters:
I20260929 17:47:44.616585 4968 leader_election.cc:288] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:44.616669 4973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:44.616843 4973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 31fbb737c0a240cda15e2a43c1f18ba6, State: Running, Role: LEADER
I20260929 17:47:44.617137 4973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38313 } }
I20260929 17:47:44.617851 4968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:44.618297 4973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31fbb737c0a240cda15e2a43c1f18ba6. Latest consensus state: current_term: 1 leader_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38313 } } }
I20260929 17:47:44.618407 4973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:44.618803 4974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fbb737c0a240cda15e2a43c1f18ba6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38313 } } }
I20260929 17:47:44.618970 4974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6 [sys.catalog]: This master's current role is: LEADER
W20260929 17:47:44.622414 4988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:47:44.622494 4988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:47:44.622550 4978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:44.623231 4978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:44.628345 4978 catalog_manager.cc:1347] Generated new cluster ID: fee000c3bff7405ab95d2e67e9e7896f
I20260929 17:47:44.628401 4978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:44.639276 4978 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:44.640535 4978 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:44.645380 4978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: Generated new TSK 0
I20260929 17:47:44.646097 4978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:47:44.672780 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38313
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:44.784323 4994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:44.784641 4994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:44.784700 4994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:44.788901 4994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:44.788966 4994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:44.789008 4994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:47:44.789055 4994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:47:44.794499 4994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:38313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:44.795603 4994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:44.796664 4994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:47:44.806232 4994 server_base.cc:1034] running on GCE node
W20260929 17:47:44.807132 4999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:44.807381 5002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:44.807821 5000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:44.808205 4994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:44.808851 4994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:44.810009 4994 hybrid_clock.cc:648] HybridClock initialized: now 1790704064809993 us; error 29 us; skew 500 ppm
Sep 29 17:47:44 dist-test-slave-wbpt krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704064, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:44.813225 4994 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:47:44.814137 4994 webserver.cc:460] Webserver started at http://127.3.183.193:46431/ using document root <none> and password file <none>
I20260929 17:47:44.814699 4994 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:44.814769 4994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:44.814953 4994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:44.817143 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "9bc93b03050c4c33b73d632503742b70"
format_stamp: "Formatted at 2026-09-29 17:47:44 on dist-test-slave-wbpt"
server_key: "d8279cd7bea6e4722d5ac0d63c539561"
server_key_iv: "1a400ac173567893c0e597a1f19144d6"
server_key_version: "encryptionkey@0"
I20260929 17:47:44.817745 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "9bc93b03050c4c33b73d632503742b70"
format_stamp: "Formatted at 2026-09-29 17:47:44 on dist-test-slave-wbpt"
server_key: "d8279cd7bea6e4722d5ac0d63c539561"
server_key_iv: "1a400ac173567893c0e597a1f19144d6"
server_key_version: "encryptionkey@0"
I20260929 17:47:44.821751 4994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:47:44.824124 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:44.825311 4994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:47:44.825415 4994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "9bc93b03050c4c33b73d632503742b70"
format_stamp: "Formatted at 2026-09-29 17:47:44 on dist-test-slave-wbpt"
server_key: "d8279cd7bea6e4722d5ac0d63c539561"
server_key_iv: "1a400ac173567893c0e597a1f19144d6"
server_key_version: "encryptionkey@0"
I20260929 17:47:44.825503 4994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:44.847700 4994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:44.850596 4994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:44.850772 4994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:44.852190 4994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:44.852240 4994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:44.852290 4994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:44.852319 4994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:47:44 dist-test-slave-wbpt krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704064, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:44.862641 4994 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:40713
I20260929 17:47:44.862756 5124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:40713 every 8 connection(s)
I20260929 17:47:44.863494 4994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:47:44.865079 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:44.852982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:55771 (local address 127.3.183.254:38313)
0929 17:47:44.853091 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:44.853094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:44.854228 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 17:47:44.855253 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:44.855256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:44.855315 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:44.855372 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:44.856769 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.857410 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.858490 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.858778 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.858881 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:44.861716 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:44.861720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:44.861722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:44.861748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:44.863129 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:44.863586 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:44.863589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:44.863591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:44.863666 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:44.864038 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:44.864041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:44.864043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:44.864268 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:44.864373 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:44.864862 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 17:47:44.864961 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:47:44.865782 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:44.853534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38313 (local address 127.3.183.193:55771)
0929 17:47:44.854076 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:44.854116 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:44.855009 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:44.855758 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:44.855764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:44.856120 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:44.856630 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:44.856649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.857531 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.857535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:44.858374 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:44.858382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.858937 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.858942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:44.859154 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:44.859805 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:44.859825 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:44.861560 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:44.863229 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:44.863235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:44.863249 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:44.863497 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:44.863746 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:44.863749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:44.863751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:44.863933 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:44.864390 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:44.864395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:44.864655 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 17:47:44.864797 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 17:47:44.865036 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":158,"threads_started":1}
I20260929 17:47:44.866909 5125 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38313
I20260929 17:47:44.867043 5125 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:44.867611 5125 heartbeater.cc:511] Master 127.3.183.254:38313 requested a full tablet report, sending...
I20260929 17:47:44.869159 4922 ts_manager.cc:194] Registered new tserver with Master: 9bc93b03050c4c33b73d632503742b70 (127.3.183.193:40713)
I20260929 17:47:44.869588 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 4994
I20260929 17:47:44.869689 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:47:44.869972 3807 external_mini_cluster.cc:1428] Setting key f20db6fd948cce580770eafc1679bf4b
I20260929 17:47:44.870522 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:55771
I20260929 17:47:44.876832 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:44.869055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38313 (local address 127.3.183.193:60569)
0929 17:47:44.869362 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:44.869374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:44.869458 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:44.869697 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:44.869700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:44.869891 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:44.869998 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:44.870004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.871140 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.871142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:44.872041 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:44.872052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.872614 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.872617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:44.872706 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:44.873210 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:44.873221 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:44.873535 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:44.875209 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:44.875212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:44.875213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:44.875411 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:44.875897 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:44.875900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:44.875901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:44.875978 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:44.876398 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:44.876401 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:44.876487 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 17:47:44.876580 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 17:47:44.876685 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":7,"thread_start_us":87,"threads_started":1}
I20260929 17:47:44.877053 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:44.869136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:60569 (local address 127.3.183.254:38313)
0929 17:47:44.869271 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:44.869275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:44.869440 (+ 165us) server_negotiation.cc:407] Connection header received
0929 17:47:44.869537 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:44.869540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:44.869579 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:44.869637 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:44.870131 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.870761 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.872154 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:44.872493 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:44.872609 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:44.873676 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:44.873680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:44.873683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:44.873707 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:44.875101 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:44.875562 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:44.875566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:44.875568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:44.875809 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:44.876061 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:44.876064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:44.876066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:44.876274 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:44.876376 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:44.876653 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 17:47:44.876771 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:47:44.894999 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38313
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:45.001561 5134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:45.001807 5134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:45.001849 5134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:45.005827 5134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:45.005882 5134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:45.005918 5134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:47:45.005954 5134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:47:45.011057 5134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:38313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:45.012085 5134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:45.013051 5134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:45.022500 5139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:45.022500 5140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:45.022541 5142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:45.022841 5134 server_base.cc:1034] running on GCE node
I20260929 17:47:45.023175 5134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:45.023765 5134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:45.024948 5134 hybrid_clock.cc:648] HybridClock initialized: now 1790704065024931 us; error 31 us; skew 500 ppm
Sep 29 17:47:45 dist-test-slave-wbpt krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:45.027526 5134 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:47:45.028473 5134 webserver.cc:460] Webserver started at http://127.3.183.194:32877/ using document root <none> and password file <none>
I20260929 17:47:45.029011 5134 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:45.029073 5134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:45.029253 5134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:45.031275 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "d1638062348a466a8dc5154377cfbd5e"
format_stamp: "Formatted at 2026-09-29 17:47:45 on dist-test-slave-wbpt"
server_key: "7221be9f82ca487f7446e70081411264"
server_key_iv: "052839e554457b16bce68b607d449945"
server_key_version: "encryptionkey@0"
I20260929 17:47:45.031838 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "d1638062348a466a8dc5154377cfbd5e"
format_stamp: "Formatted at 2026-09-29 17:47:45 on dist-test-slave-wbpt"
server_key: "7221be9f82ca487f7446e70081411264"
server_key_iv: "052839e554457b16bce68b607d449945"
server_key_version: "encryptionkey@0"
I20260929 17:47:45.035351 5134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:47:45.037767 5149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:45.038977 5134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:45.039091 5134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "d1638062348a466a8dc5154377cfbd5e"
format_stamp: "Formatted at 2026-09-29 17:47:45 on dist-test-slave-wbpt"
server_key: "7221be9f82ca487f7446e70081411264"
server_key_iv: "052839e554457b16bce68b607d449945"
server_key_version: "encryptionkey@0"
I20260929 17:47:45.039186 5134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:45.052726 5134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:45.055754 5134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:45.055967 5134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:45.057420 5134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:45.057472 5134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:45.057525 5134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:45.057554 5134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:47:45 dist-test-slave-wbpt krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:45.070094 5134 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:38567
I20260929 17:47:45.070524 5264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:38567 every 8 connection(s)
I20260929 17:47:45.071194 5134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:47:45.071246 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.058250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:39097 (local address 127.3.183.254:38313)
0929 17:47:45.058367 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.058371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.059400 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 17:47:45.060483 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.060487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.060550 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.060642 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:45.061750 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.062333 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.063536 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.063754 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.063868 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.067244 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:45.067249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:45.067252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:45.067285 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:45.069144 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.069602 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.069606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.069608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.069710 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.070068 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.070071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.070073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.070294 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:45.070383 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.070990 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.071124 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:47:45.072284 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.058782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38313 (local address 127.3.183.194:39097)
0929 17:47:45.059254 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.059293 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.060259 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.060805 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.060814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.061144 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:45.061614 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.061632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.062457 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.062466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.063399 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.063417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.063915 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.063919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.064078 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.064641 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:45.064660 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:45.067012 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:45.069252 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.069257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.069268 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.069505 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.069799 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.069802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.069804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.069969 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.070433 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:45.070439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:45.070740 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.070942 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.071257 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260929 17:47:45.073411 5265 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38313
I20260929 17:47:45.073556 5265 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:45.074113 5265 heartbeater.cc:511] Master 127.3.183.254:38313 requested a full tablet report, sending...
I20260929 17:47:45.075201 4922 ts_manager.cc:194] Registered new tserver with Master: d1638062348a466a8dc5154377cfbd5e (127.3.183.194:38567)
I20260929 17:47:45.075937 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:39097
I20260929 17:47:45.080643 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 5134
I20260929 17:47:45.080750 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:47:45.080986 3807 external_mini_cluster.cc:1428] Setting key 580b94b5a8e062555e6ccd2aab6b384e
I20260929 17:47:45.082456 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.075600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38313 (local address 127.3.183.194:36137)
0929 17:47:45.075933 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.075945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.076033 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.076260 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.076263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.076472 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:45.076615 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.076622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.077573 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.077575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.078119 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.078127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.078517 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.078520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.078590 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.079015 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:45.079025 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:45.079333 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:45.080850 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.080853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.080855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.081041 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.081323 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.081326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.081328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.081415 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.081878 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:45.081880 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:45.081985 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.082184 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.082310 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260929 17:47:45.082680 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.075690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:36137 (local address 127.3.183.254:38313)
0929 17:47:45.075810 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.075814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.076006 (+ 192us) server_negotiation.cc:407] Connection header received
0929 17:47:45.076103 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.076106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.076158 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.076225 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:45.076740 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.077471 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.078214 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.078415 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.078490 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.079524 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:45.079527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:45.079529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:45.079550 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:45.080750 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.081135 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.081139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.081140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.081239 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.081490 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.081492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.081493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.081749 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:45.081886 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.082415 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.082526 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:47:45.104424 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38313
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:45.211540 5274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:45.211807 5274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:45.211853 5274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:45.215719 5274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:45.215767 5274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:45.215822 5274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:47:45.215857 5274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:47:45.221225 5274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:38313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:45.222215 5274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:45.223181 5274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:45.232386 5282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:45.232412 5280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:45.232494 5274 server_base.cc:1034] running on GCE node
W20260929 17:47:45.232429 5279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:45.233019 5274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:45.233615 5274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:45.234795 5274 hybrid_clock.cc:648] HybridClock initialized: now 1790704065234761 us; error 45 us; skew 500 ppm
Sep 29 17:47:45 dist-test-slave-wbpt krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:45.237460 5274 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:47:45.238319 5274 webserver.cc:460] Webserver started at http://127.3.183.195:46729/ using document root <none> and password file <none>
I20260929 17:47:45.238857 5274 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:45.238922 5274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:45.239140 5274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:45.241190 5274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "496649c21caa4f07bdf2461ceb796d05"
format_stamp: "Formatted at 2026-09-29 17:47:45 on dist-test-slave-wbpt"
server_key: "539652590d613f3f093f171f55699497"
server_key_iv: "f27b46a9193e652141d901556b6e2b8c"
server_key_version: "encryptionkey@0"
I20260929 17:47:45.241828 5274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "496649c21caa4f07bdf2461ceb796d05"
format_stamp: "Formatted at 2026-09-29 17:47:45 on dist-test-slave-wbpt"
server_key: "539652590d613f3f093f171f55699497"
server_key_iv: "f27b46a9193e652141d901556b6e2b8c"
server_key_version: "encryptionkey@0"
I20260929 17:47:45.245957 5274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:47:45.248296 5289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:45.249511 5274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:45.249604 5274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "496649c21caa4f07bdf2461ceb796d05"
format_stamp: "Formatted at 2026-09-29 17:47:45 on dist-test-slave-wbpt"
server_key: "539652590d613f3f093f171f55699497"
server_key_iv: "f27b46a9193e652141d901556b6e2b8c"
server_key_version: "encryptionkey@0"
I20260929 17:47:45.249698 5274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:45.259261 5274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:45.261924 5274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:45.262066 5274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:45.263382 5274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:45.263432 5274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:45.263470 5274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:45.263495 5274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:47:45 dist-test-slave-wbpt krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704065, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:45.273703 5274 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:34813
I20260929 17:47:45.274137 5404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:34813 every 8 connection(s)
I20260929 17:47:45.274688 5274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:47:45.277283 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.264090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:46225 (local address 127.3.183.254:38313)
0929 17:47:45.264227 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.264231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.265121 (+ 890us) server_negotiation.cc:407] Connection header received
0929 17:47:45.266356 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.266360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.266430 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.266533 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:45.268054 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.268740 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.270451 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.270733 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.270813 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.273765 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:45.273769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:45.273772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:45.273801 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:45.275338 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.275973 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.275976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.275977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.276064 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.276349 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.276351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.276352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.276530 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:45.276601 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.277035 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.277133 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:47:45.278138 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.264482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38313 (local address 127.3.183.195:46225)
0929 17:47:45.264950 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.265006 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.266035 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.266742 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.266752 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.267206 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:45.267888 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.267910 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.268865 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.268869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.270333 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.270344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.270874 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.270878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.271128 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.271830 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:45.271850 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:45.273556 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:45.275506 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.275513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.275530 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.275868 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.276145 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.276147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.276149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.276276 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.276602 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:45.276607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:45.276830 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.276999 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.277265 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
I20260929 17:47:45.279278 5405 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38313
I20260929 17:47:45.279414 5405 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:45.279829 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 5274
I20260929 17:47:45.279932 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:47:45.279946 5405 heartbeater.cc:511] Master 127.3.183.254:38313 requested a full tablet report, sending...
I20260929 17:47:45.280239 3807 external_mini_cluster.cc:1428] Setting key 79bc7873274b151523153d357f43bebd
I20260929 17:47:45.281041 4921 ts_manager.cc:194] Registered new tserver with Master: 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195:34813)
I20260929 17:47:45.281713 4921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:46225
I20260929 17:47:45.283399 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:47:45.288597 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.281601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38313 (local address 127.3.183.195:34991)
0929 17:47:45.281871 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.281885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.281996 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.282238 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.282241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.282439 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:45.282579 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.282586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.283554 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.283557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.284214 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.284223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.284744 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.284747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.284824 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.285299 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:45.285310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:45.285596 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:45.287080 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.287083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.287085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.287337 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.287655 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.287657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.287659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.287753 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.288111 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:45.288114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:45.288217 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.288323 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.288461 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":7,"thread_start_us":96,"threads_started":1}
I20260929 17:47:45.288692 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.281739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:34991 (local address 127.3.183.254:38313)
0929 17:47:45.281853 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.281857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.281952 (+ 95us) server_negotiation.cc:407] Connection header received
0929 17:47:45.282108 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.282112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.282152 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.282212 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:45.282682 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.283407 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.284324 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.284647 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.284743 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.285729 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:45.285733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:45.285735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:45.285758 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:45.286976 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.287418 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.287420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.287422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.287564 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.287866 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.287868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.287869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.288029 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:45.288101 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.288372 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.288547 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 17:47:45 dist-test-slave-wbpt krb5kdc[4877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:47:45 dist-test-slave-wbpt krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:45.308148 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.300676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.3.183.254:38313)
0929 17:47:45.300795 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.300798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.300809 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:47:45.300849 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.300851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.300886 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.300952 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:45.301610 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.302124 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.302932 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.303142 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.303226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.305430 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:45.305433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:45.305435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:45.305458 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:45.306681 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.307038 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.307042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.307043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.307115 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.307351 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.307354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.307356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.307521 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:45.307645 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.307894 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.307996 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:47:45.310740 4921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:47:45.313755 4921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:45.327206 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.321712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:38567 (local address 127.0.0.1:37762)
0929 17:47:45.322109 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.322157 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.322277 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.323019 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.323025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.323048 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:45.323176 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.323183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.324413 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.324416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.325641 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.325651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.326691 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.326694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.326799 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.326840 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.326945 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.327059 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":79,"threads_started":1}
I20260929 17:47:45.327370 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.321773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34813 (local address 127.0.0.1:58446)
0929 17:47:45.322395 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.322407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.322483 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.323271 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.323274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.323291 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:45.323385 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.323391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.324858 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.324861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.325974 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.325985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.326950 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.326954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.327039 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.327061 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.327117 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.327194 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":25,"thread_start_us":109,"threads_started":1}
I20260929 17:47:45.327523 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.322435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:40713 (local address 127.0.0.1:38760)
0929 17:47:45.323231 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.323243 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.323305 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.324010 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.324013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.324031 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:45.324112 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.324117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.325215 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.325218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.326184 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.326193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.327167 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.327170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.327258 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.327279 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.327345 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.327408 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260929 17:47:45.327730 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.321855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.3.183.194:38567)
0929 17:47:45.322245 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.322256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.322280 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:47:45.322477 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.322483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.322707 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.322855 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:45.323301 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.324325 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.325773 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.326573 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.326682 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.326842 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.327502 (+ 660us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.327591 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":97,"threads_started":1}
I20260929 17:47:45.327867 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.322513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.3.183.195:34813)
0929 17:47:45.322841 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.322852 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.322874 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:47:45.322941 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.322945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.323122 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.323515 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:45.323643 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.324745 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.326097 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.326847 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.326935 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.327084 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.327142 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.327226 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
I20260929 17:47:45.327899 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.322888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.3.183.193:40713)
0929 17:47:45.323546 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.323556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.323577 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:47:45.323645 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.323649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.323861 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.323997 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:45.324229 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.325112 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.326291 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.326875 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.326964 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.327100 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.327668 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.327750 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":95,"threads_started":1}
I20260929 17:47:45.330348 5199 tablet_service.cc:1467] Processing CreateTablet for tablet b630d3e61e554e2e9a3bb5a8c3983fb5 (DEFAULT_TABLE table=test-table [id=a657a417d95348969d41aea902b89315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:45.330402 5059 tablet_service.cc:1467] Processing CreateTablet for tablet b630d3e61e554e2e9a3bb5a8c3983fb5 (DEFAULT_TABLE table=test-table [id=a657a417d95348969d41aea902b89315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:45.330402 5339 tablet_service.cc:1467] Processing CreateTablet for tablet b630d3e61e554e2e9a3bb5a8c3983fb5 (DEFAULT_TABLE table=test-table [id=a657a417d95348969d41aea902b89315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:45.331260 5199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b630d3e61e554e2e9a3bb5a8c3983fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.331367 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b630d3e61e554e2e9a3bb5a8c3983fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.331374 5339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b630d3e61e554e2e9a3bb5a8c3983fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.338423 5428 tablet_bootstrap.cc:492] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Bootstrap starting.
I20260929 17:47:45.339233 5429 tablet_bootstrap.cc:492] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Bootstrap starting.
I20260929 17:47:45.339504 5430 tablet_bootstrap.cc:492] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Bootstrap starting.
I20260929 17:47:45.340564 5428 tablet_bootstrap.cc:654] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.341226 5428 log.cc:824] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:45.341354 5429 tablet_bootstrap.cc:654] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.341353 5430 tablet_bootstrap.cc:654] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.342161 5429 log.cc:824] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:45.342168 5430 log.cc:824] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:45.343096 5428 tablet_bootstrap.cc:492] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: No bootstrap required, opened a new log
I20260929 17:47:45.343281 5428 ts_tablet_manager.cc:1397] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:47:45.344473 5430 tablet_bootstrap.cc:492] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: No bootstrap required, opened a new log
I20260929 17:47:45.344650 5430 ts_tablet_manager.cc:1397] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:47:45.344991 5429 tablet_bootstrap.cc:492] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: No bootstrap required, opened a new log
I20260929 17:47:45.345150 5429 ts_tablet_manager.cc:1397] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 17:47:45.347056 5428 raft_consensus.cc:348] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.347317 5428 raft_consensus.cc:374] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.347378 5428 raft_consensus.cc:729] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1638062348a466a8dc5154377cfbd5e, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.347836 5428 consensus_queue.cc:260] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.347915 5429 raft_consensus.cc:348] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.348152 5429 raft_consensus.cc:374] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.348218 5429 raft_consensus.cc:729] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc93b03050c4c33b73d632503742b70, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.348284 5430 raft_consensus.cc:348] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.348553 5430 raft_consensus.cc:374] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.348598 5430 raft_consensus.cc:729] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496649c21caa4f07bdf2461ceb796d05, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.348600 5265 heartbeater.cc:503] Master 127.3.183.254:38313 was elected leader, sending a full tablet report...
I20260929 17:47:45.348616 5429 consensus_queue.cc:260] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.348839 5428 ts_tablet_manager.cc:1428] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:47:45.349215 5430 consensus_queue.cc:260] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.349443 5429 ts_tablet_manager.cc:1428] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 17:47:45.349745 5125 heartbeater.cc:503] Master 127.3.183.254:38313 was elected leader, sending a full tablet report...
I20260929 17:47:45.350075 5430 ts_tablet_manager.cc:1428] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:47:45.350368 5405 heartbeater.cc:503] Master 127.3.183.254:38313 was elected leader, sending a full tablet report...
W20260929 17:47:45.364784 5126 tablet.cc:2369] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:47:45.441061 5435 raft_consensus.cc:482] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:45.441258 5435 raft_consensus.cc:504] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.442380 5435 leader_election.cc:288] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1638062348a466a8dc5154377cfbd5e (127.3.183.194:38567), 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195:34813)
I20260929 17:47:45.447367 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.442741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:37013 (local address 127.3.183.194:38567)
0929 17:47:45.442878 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.442882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.443044 (+ 162us) server_negotiation.cc:407] Connection header received
0929 17:47:45.443129 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.443131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.443172 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.443250 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:45.443619 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.444437 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.446118 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.446721 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.446833 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.446884 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.447174 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.447238 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:47:45.447367 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.442745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:36709 (local address 127.3.183.195:34813)
0929 17:47:45.442922 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.442927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.442944 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:47:45.443108 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.443112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.443173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.443250 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:45.443603 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.444438 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.446118 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.446720 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.446833 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.446884 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.447176 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.447241 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 17:47:45.447402 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.442614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:38567 (local address 127.3.183.193:37013)
0929 17:47:45.442973 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.442983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.443058 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.443265 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.443268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.443307 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:45.443483 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.443490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.444565 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.444568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.446000 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.446012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.446839 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.446843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.447076 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.447094 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.447154 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.447232 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":310,"spinlock_wait_cycles":30464,"thread_start_us":87,"threads_started":1}
I20260929 17:47:45.447405 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.442607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34813 (local address 127.3.183.193:36709)
0929 17:47:45.442886 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.442901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.443006 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.443266 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.443268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.443308 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:45.443489 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.443495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.444554 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.444558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.445993 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.446019 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.446839 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.446843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.447026 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.447068 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.447144 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.447232 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":138,"spinlock_wait_cycles":3200,"thread_start_us":67,"threads_started":1}
I20260929 17:47:45.448030 5359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b630d3e61e554e2e9a3bb5a8c3983fb5" candidate_uuid: "9bc93b03050c4c33b73d632503742b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496649c21caa4f07bdf2461ceb796d05" is_pre_election: true
I20260929 17:47:45.448071 5219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b630d3e61e554e2e9a3bb5a8c3983fb5" candidate_uuid: "9bc93b03050c4c33b73d632503742b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1638062348a466a8dc5154377cfbd5e" is_pre_election: true
I20260929 17:47:45.448342 5219 raft_consensus.cc:2454] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc93b03050c4c33b73d632503742b70 in term 0.
I20260929 17:47:45.448343 5359 raft_consensus.cc:2454] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc93b03050c4c33b73d632503742b70 in term 0.
I20260929 17:47:45.448812 5013 leader_election.cc:302] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc93b03050c4c33b73d632503742b70, d1638062348a466a8dc5154377cfbd5e; no voters:
I20260929 17:47:45.449084 5435 raft_consensus.cc:2781] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:45.449165 5435 raft_consensus.cc:482] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:45.449198 5435 raft_consensus.cc:3037] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:45.450201 5435 raft_consensus.cc:504] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.450515 5435 leader_election.cc:288] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [CANDIDATE]: Term 1 election: Requested vote from peers d1638062348a466a8dc5154377cfbd5e (127.3.183.194:38567), 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195:34813)
I20260929 17:47:45.450860 5359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b630d3e61e554e2e9a3bb5a8c3983fb5" candidate_uuid: "9bc93b03050c4c33b73d632503742b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496649c21caa4f07bdf2461ceb796d05"
I20260929 17:47:45.450860 5219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b630d3e61e554e2e9a3bb5a8c3983fb5" candidate_uuid: "9bc93b03050c4c33b73d632503742b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1638062348a466a8dc5154377cfbd5e"
I20260929 17:47:45.450981 5219 raft_consensus.cc:3037] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:45.450981 5359 raft_consensus.cc:3037] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:45.452347 5359 raft_consensus.cc:2454] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc93b03050c4c33b73d632503742b70 in term 1.
I20260929 17:47:45.452688 5219 raft_consensus.cc:2454] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc93b03050c4c33b73d632503742b70 in term 1.
I20260929 17:47:45.452708 5010 leader_election.cc:302] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496649c21caa4f07bdf2461ceb796d05, 9bc93b03050c4c33b73d632503742b70; no voters:
I20260929 17:47:45.452893 5435 raft_consensus.cc:2781] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:45.453082 5435 raft_consensus.cc:686] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc93b03050c4c33b73d632503742b70, State: Running, Role: LEADER
I20260929 17:47:45.453459 5435 consensus_queue.cc:237] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } }
I20260929 17:47:45.456456 4921 catalog_manager.cc:5495] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc93b03050c4c33b73d632503742b70 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "9bc93b03050c4c33b73d632503742b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } health_report { overall_health: HEALTHY } } }
I20260929 17:47:45.483359 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.479857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.3.183.193:40713)
0929 17:47:45.479980 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.479984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.479998 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:47:45.480120 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.480122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.480174 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.480260 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:45.480464 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.480987 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.481804 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.482034 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.482111 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.482425 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:45.482866 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:45.482983 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.483190 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.483249 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:47:45.489152 5359 raft_consensus.cc:1261] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Refusing update from remote peer 9bc93b03050c4c33b73d632503742b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:45.489856 5439 consensus_queue.cc:1035] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:45.491299 5219 raft_consensus.cc:1261] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Refusing update from remote peer 9bc93b03050c4c33b73d632503742b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:45.491865 5439 consensus_queue.cc:1035] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:45.498773 5443 mvcc.cc:205] Tried to move back new op lower bound from 7334723852235784192 to 7334723852099407872. Current Snapshot: MvccSnapshot[applied={T|T < 7334723852235784192}]
I20260929 17:47:45.499763 5445 mvcc.cc:205] Tried to move back new op lower bound from 7334723852235784192 to 7334723852099407872. Current Snapshot: MvccSnapshot[applied={T|T < 7334723852235784192}]
I20260929 17:47:45.510138 4920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:45.510404 4920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:45.513016 4920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: Sending DeleteTablet for 3 replicas of tablet b630d3e61e554e2e9a3bb5a8c3983fb5
I20260929 17:47:45.513814 5199 tablet_service.cc:1514] Processing DeleteTablet for tablet b630d3e61e554e2e9a3bb5a8c3983fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:45 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:37762
I20260929 17:47:45.513936 5059 tablet_service.cc:1514] Processing DeleteTablet for tablet b630d3e61e554e2e9a3bb5a8c3983fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:45 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:38760
I20260929 17:47:45.514038 5339 tablet_service.cc:1514] Processing DeleteTablet for tablet b630d3e61e554e2e9a3bb5a8c3983fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:45 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:58446
I20260929 17:47:45.514472 5453 tablet_replica.cc:330] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: stopping tablet replica
I20260929 17:47:45.514472 5452 tablet_replica.cc:330] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: stopping tablet replica
I20260929 17:47:45.514794 5452 raft_consensus.cc:2229] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:45.514794 5453 raft_consensus.cc:2229] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:47:45.515260 5453 raft_consensus.cc:2258] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:45.515219 4920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:47:45.515607 4920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:45.516358 5452 raft_consensus.cc:2258] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:45.516710 5454 tablet_replica.cc:330] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: stopping tablet replica
I20260929 17:47:45.516741 5453 ts_tablet_manager.cc:1905] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:45.517023 5454 raft_consensus.cc:2229] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:45.520356 5452 ts_tablet_manager.cc:1905] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:45.520764 5059 tablet_service.cc:1467] Processing CreateTablet for tablet 64d23d2948fa422498f05a36f51c92a1 (DEFAULT_TABLE table=test-table [id=b92f3ae0b5bf4094ad50bbeff571154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:45.520913 5199 tablet_service.cc:1467] Processing CreateTablet for tablet 64d23d2948fa422498f05a36f51c92a1 (DEFAULT_TABLE table=test-table [id=b92f3ae0b5bf4094ad50bbeff571154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:45.521009 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d23d2948fa422498f05a36f51c92a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.521139 5199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d23d2948fa422498f05a36f51c92a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.523561 5429 tablet_bootstrap.cc:492] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: Bootstrap starting.
I20260929 17:47:45.523819 5428 tablet_bootstrap.cc:492] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Bootstrap starting.
I20260929 17:47:45.524307 5339 tablet_service.cc:1467] Processing CreateTablet for tablet 64d23d2948fa422498f05a36f51c92a1 (DEFAULT_TABLE table=test-table [id=b92f3ae0b5bf4094ad50bbeff571154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:45.524534 5429 tablet_bootstrap.cc:654] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.524573 5339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d23d2948fa422498f05a36f51c92a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.525321 5428 tablet_bootstrap.cc:654] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.526944 5428 tablet_bootstrap.cc:492] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: No bootstrap required, opened a new log
I20260929 17:47:45.527031 5428 ts_tablet_manager.cc:1397] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:47:45.527243 5430 tablet_bootstrap.cc:492] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Bootstrap starting.
I20260929 17:47:45.527644 5428 raft_consensus.cc:348] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.527750 5428 raft_consensus.cc:374] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.527799 5428 raft_consensus.cc:729] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1638062348a466a8dc5154377cfbd5e, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.527941 5428 consensus_queue.cc:260] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.528277 5428 ts_tablet_manager.cc:1428] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:47:45.528604 5453 ts_tablet_manager.cc:1918] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:47:45.528700 5453 log.cc:1196] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/b630d3e61e554e2e9a3bb5a8c3983fb5
I20260929 17:47:45.528833 5430 tablet_bootstrap.cc:654] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.529026 5453 ts_tablet_manager.cc:1939] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 9bc93b03050c4c33b73d632503742b70: Deleting consensus metadata
I20260929 17:47:45.530251 4909 catalog_manager.cc:4841] TS 9bc93b03050c4c33b73d632503742b70 (127.3.183.193:40713): tablet b630d3e61e554e2e9a3bb5a8c3983fb5 (table test-table [id=a657a417d95348969d41aea902b89315]) successfully deleted
I20260929 17:47:45.530776 5429 tablet_bootstrap.cc:492] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: No bootstrap required, opened a new log
I20260929 17:47:45.530867 5429 ts_tablet_manager.cc:1397] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 17:47:45.517541 5454 raft_consensus.cc:2258] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:45.531404 5429 raft_consensus.cc:348] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.531579 5429 raft_consensus.cc:374] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.531623 5429 raft_consensus.cc:729] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc93b03050c4c33b73d632503742b70, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.531751 5429 consensus_queue.cc:260] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.532030 5429 ts_tablet_manager.cc:1428] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:47:45.534006 5454 ts_tablet_manager.cc:1905] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:45.535725 5452 ts_tablet_manager.cc:1918] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:47:45.535857 5452 log.cc:1196] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/b630d3e61e554e2e9a3bb5a8c3983fb5
I20260929 17:47:45.536531 5452 ts_tablet_manager.cc:1939] T b630d3e61e554e2e9a3bb5a8c3983fb5 P d1638062348a466a8dc5154377cfbd5e: Deleting consensus metadata
I20260929 17:47:45.536880 5454 ts_tablet_manager.cc:1918] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:47:45.536978 5454 log.cc:1196] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/b630d3e61e554e2e9a3bb5a8c3983fb5
I20260929 17:47:45.537292 5454 ts_tablet_manager.cc:1939] T b630d3e61e554e2e9a3bb5a8c3983fb5 P 496649c21caa4f07bdf2461ceb796d05: Deleting consensus metadata
I20260929 17:47:45.537793 4909 catalog_manager.cc:4841] TS d1638062348a466a8dc5154377cfbd5e (127.3.183.194:38567): tablet b630d3e61e554e2e9a3bb5a8c3983fb5 (table test-table [id=a657a417d95348969d41aea902b89315]) successfully deleted
I20260929 17:47:45.538316 5430 tablet_bootstrap.cc:492] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: No bootstrap required, opened a new log
I20260929 17:47:45.538396 5430 ts_tablet_manager.cc:1397] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260929 17:47:45.538389 4906 catalog_manager.cc:4841] TS 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195:34813): tablet b630d3e61e554e2e9a3bb5a8c3983fb5 (table test-table [id=a657a417d95348969d41aea902b89315]) successfully deleted
I20260929 17:47:45.538816 5430 raft_consensus.cc:348] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.538913 5430 raft_consensus.cc:374] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.538940 5430 raft_consensus.cc:729] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496649c21caa4f07bdf2461ceb796d05, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.539062 5430 consensus_queue.cc:260] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.539297 5430 ts_tablet_manager.cc:1428] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260929 17:47:45.572896 5266 tablet.cc:2369] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:47:45.775848 5406 tablet.cc:2369] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:47:45.835345 5436 raft_consensus.cc:482] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:45.835515 5436 raft_consensus.cc:504] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.836648 5436 leader_election.cc:288] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc93b03050c4c33b73d632503742b70 (127.3.183.193:40713), d1638062348a466a8dc5154377cfbd5e (127.3.183.194:38567)
I20260929 17:47:45.840814 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.836964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:40713 (local address 127.3.183.195:45765)
0929 17:47:45.837206 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.837221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.837352 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.837705 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.837708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.837731 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:45.837868 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.837875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.838747 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.838750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.839685 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.839695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.840348 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.840352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.840460 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.840487 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.840566 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.840642 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":52,"thread_start_us":65,"threads_started":1}
I20260929 17:47:45.840790 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.837058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:45765 (local address 127.3.183.193:40713)
0929 17:47:45.837185 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.837195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.837340 (+ 145us) server_negotiation.cc:407] Connection header received
0929 17:47:45.837504 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.837508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.837571 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.837681 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:45.837990 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.838632 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.839825 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.840230 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.840350 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.840413 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.840584 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.840659 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:47:45.840977 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.837544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:43537 (local address 127.3.183.194:38567)
0929 17:47:45.837655 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.837659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.837721 (+ 62us) server_negotiation.cc:407] Connection header received
0929 17:47:45.837852 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.837855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.837898 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.837967 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:45.838276 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.838782 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.839999 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.840396 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.840499 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.840569 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.840773 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.840856 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 17:47:45.841375 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.837389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:38567 (local address 127.3.183.195:43537)
0929 17:47:45.837634 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.837650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.837755 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.837994 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.837997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.838015 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:45.838122 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.838129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.838887 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.838891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.839887 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.839897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.840499 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.840503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.840634 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.840656 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.841104 (+ 448us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.841188 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":54,"spinlock_wait_cycles":1792,"thread_start_us":64,"threads_started":1}
I20260929 17:47:45.841379 5079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d23d2948fa422498f05a36f51c92a1" candidate_uuid: "496649c21caa4f07bdf2461ceb796d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc93b03050c4c33b73d632503742b70" is_pre_election: true
I20260929 17:47:45.841624 5079 raft_consensus.cc:2454] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496649c21caa4f07bdf2461ceb796d05 in term 0.
I20260929 17:47:45.841884 5219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d23d2948fa422498f05a36f51c92a1" candidate_uuid: "496649c21caa4f07bdf2461ceb796d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1638062348a466a8dc5154377cfbd5e" is_pre_election: true
I20260929 17:47:45.842046 5219 raft_consensus.cc:2454] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496649c21caa4f07bdf2461ceb796d05 in term 0.
I20260929 17:47:45.842097 5293 leader_election.cc:302] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496649c21caa4f07bdf2461ceb796d05, 9bc93b03050c4c33b73d632503742b70; no voters:
I20260929 17:47:45.842394 5436 raft_consensus.cc:2781] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:45.842458 5436 raft_consensus.cc:482] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:45.842510 5436 raft_consensus.cc:3037] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:45.843873 5436 raft_consensus.cc:504] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.844221 5436 leader_election.cc:288] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc93b03050c4c33b73d632503742b70 (127.3.183.193:40713), d1638062348a466a8dc5154377cfbd5e (127.3.183.194:38567)
I20260929 17:47:45.844565 5079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d23d2948fa422498f05a36f51c92a1" candidate_uuid: "496649c21caa4f07bdf2461ceb796d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc93b03050c4c33b73d632503742b70"
I20260929 17:47:45.844630 5219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d23d2948fa422498f05a36f51c92a1" candidate_uuid: "496649c21caa4f07bdf2461ceb796d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1638062348a466a8dc5154377cfbd5e"
I20260929 17:47:45.844679 5079 raft_consensus.cc:3037] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:45.844743 5219 raft_consensus.cc:3037] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:45.845690 5219 raft_consensus.cc:2454] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 496649c21caa4f07bdf2461ceb796d05 in term 1.
I20260929 17:47:45.845743 5079 raft_consensus.cc:2454] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 496649c21caa4f07bdf2461ceb796d05 in term 1.
I20260929 17:47:45.846060 5293 leader_election.cc:302] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496649c21caa4f07bdf2461ceb796d05, 9bc93b03050c4c33b73d632503742b70; no voters:
I20260929 17:47:45.846242 5436 raft_consensus.cc:2781] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:45.846428 5436 raft_consensus.cc:686] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 1 LEADER]: Becoming Leader. State: Replica: 496649c21caa4f07bdf2461ceb796d05, State: Running, Role: LEADER
I20260929 17:47:45.846691 5436 consensus_queue.cc:237] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.848970 4920 catalog_manager.cc:5495] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195). New cstate: current_term: 1 leader_uuid: "496649c21caa4f07bdf2461ceb796d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } health_report { overall_health: HEALTHY } } }
Sep 29 17:47:45 dist-test-slave-wbpt krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704064, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:45.918263 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.909444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38313 (local address 127.0.0.1:55026)
0929 17:47:45.909883 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:45.909896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:45.910009 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:45.910266 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:45.910269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:45.910790 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:45.910922 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.910931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.911683 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.911687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.912257 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:45.912265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.912669 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.912672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:45.912742 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.913348 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:45.913360 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:45.914874 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:45.916734 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.916743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.916748 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.916983 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.917206 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:45.917208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:45.917209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:45.917281 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:45.917684 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:45.917690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:45.917821 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 17:47:45.917949 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 17:47:45.918087 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":184,"threads_started":1}
I20260929 17:47:45.918282 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:45.909569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.3.183.254:38313)
0929 17:47:45.909838 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:45.909848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:45.909970 (+ 122us) server_negotiation.cc:407] Connection header received
0929 17:47:45.910095 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:45.910098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:45.910136 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:45.910215 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:45.911046 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.911579 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.912366 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:45.912574 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:45.912662 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:45.915026 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:45.915030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:45.915032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:45.915053 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:45.916606 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.917064 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.917067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.917068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.917137 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:45.917354 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:45.917357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:45.917359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:45.917581 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:45.917655 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:45.918007 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 17:47:45.918145 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
I20260929 17:47:45.922209 4920 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:55026:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:47:45.922978 4920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:45.927665 5059 tablet_service.cc:1467] Processing CreateTablet for tablet 309eaa8fbe054f81948d1f08a2d97b46 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5a25996b4d042e2b09793005f80298c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:47:45.927717 5199 tablet_service.cc:1467] Processing CreateTablet for tablet 309eaa8fbe054f81948d1f08a2d97b46 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5a25996b4d042e2b09793005f80298c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:47:45.927815 5339 tablet_service.cc:1467] Processing CreateTablet for tablet 309eaa8fbe054f81948d1f08a2d97b46 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5a25996b4d042e2b09793005f80298c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:47:45.928104 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309eaa8fbe054f81948d1f08a2d97b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.928107 5199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309eaa8fbe054f81948d1f08a2d97b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.928117 5339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309eaa8fbe054f81948d1f08a2d97b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:45.930927 5428 tablet_bootstrap.cc:492] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: Bootstrap starting.
I20260929 17:47:45.931082 5429 tablet_bootstrap.cc:492] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: Bootstrap starting.
I20260929 17:47:45.931814 5429 tablet_bootstrap.cc:654] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.931938 5430 tablet_bootstrap.cc:492] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: Bootstrap starting.
I20260929 17:47:45.932300 5428 tablet_bootstrap.cc:654] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.932967 5429 tablet_bootstrap.cc:492] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: No bootstrap required, opened a new log
I20260929 17:47:45.933050 5429 ts_tablet_manager.cc:1397] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:45.933418 5430 tablet_bootstrap.cc:654] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:45.933475 5429 raft_consensus.cc:348] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.933563 5429 raft_consensus.cc:374] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.933583 5429 raft_consensus.cc:729] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc93b03050c4c33b73d632503742b70, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.933681 5429 consensus_queue.cc:260] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.933904 5429 ts_tablet_manager.cc:1428] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:47:45.933938 5448 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:45.934059 5448 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:45.934919 5428 tablet_bootstrap.cc:492] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: No bootstrap required, opened a new log
I20260929 17:47:45.934998 5428 ts_tablet_manager.cc:1397] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:47:45.935225 5430 tablet_bootstrap.cc:492] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: No bootstrap required, opened a new log
I20260929 17:47:45.935305 5430 ts_tablet_manager.cc:1397] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260929 17:47:45.935634 5428 raft_consensus.cc:348] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.935745 5428 raft_consensus.cc:374] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.935838 5428 raft_consensus.cc:729] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1638062348a466a8dc5154377cfbd5e, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.935961 5428 consensus_queue.cc:260] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.935979 5430 raft_consensus.cc:348] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.936091 5430 raft_consensus.cc:374] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:45.936136 5430 raft_consensus.cc:729] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496649c21caa4f07bdf2461ceb796d05, State: Initialized, Role: FOLLOWER
I20260929 17:47:45.936218 5428 ts_tablet_manager.cc:1428] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 17:47:45.936282 5430 consensus_queue.cc:260] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:45.936517 5436 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:45.936489 5434 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:45.936647 5434 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:45.936646 5436 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:45.936856 5430 ts_tablet_manager.cc:1428] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:46.074010 5434 raft_consensus.cc:482] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:46.074144 5434 raft_consensus.cc:504] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:46.075273 5434 leader_election.cc:288] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc93b03050c4c33b73d632503742b70 (127.3.183.193:40713), 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195:34813)
I20260929 17:47:46.080046 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.075808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:54305 (local address 127.3.183.193:40713)
0929 17:47:46.075925 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.075929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.076251 (+ 322us) server_negotiation.cc:407] Connection header received
0929 17:47:46.076430 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.076433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.076490 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.076539 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:46.076939 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.077711 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.079037 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.079418 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.079532 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.079588 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.079799 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.079884 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:47:46.080170 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.075507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34813 (local address 127.3.183.194:43949)
0929 17:47:46.075842 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:46.075858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:46.075962 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:46.076595 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:46.076598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:46.076613 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:46.076720 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.076726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.077524 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.077528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.078721 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.078731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.079672 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.079675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.079760 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.079802 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 17:47:46.079863 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:47:46.079999 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":88,"threads_started":1}
I20260929 17:47:46.080401 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.075542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:40713 (local address 127.3.183.194:54305)
0929 17:47:46.076174 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:46.076187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:46.076319 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:46.076677 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:46.076679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:46.076698 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:46.076816 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.076822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.077824 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.077827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.078931 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.078940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.079532 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.079536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.079640 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.079665 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:47:46.080211 (+ 546us) client_negotiation.cc:240] Negotiation successful
0929 17:47:46.080281 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":56,"threads_started":1}
I20260929 17:47:46.080497 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.076026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:43949 (local address 127.3.183.195:34813)
0929 17:47:46.076244 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.076255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.076268 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:47:46.076318 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.076321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.076365 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.076472 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:46.076827 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.077411 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.078848 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.079568 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.079686 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.079733 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.079878 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.079952 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260929 17:47:46.080833 5079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309eaa8fbe054f81948d1f08a2d97b46" candidate_uuid: "d1638062348a466a8dc5154377cfbd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc93b03050c4c33b73d632503742b70" is_pre_election: true
I20260929 17:47:46.080991 5359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309eaa8fbe054f81948d1f08a2d97b46" candidate_uuid: "d1638062348a466a8dc5154377cfbd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496649c21caa4f07bdf2461ceb796d05" is_pre_election: true
I20260929 17:47:46.081040 5079 raft_consensus.cc:2454] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1638062348a466a8dc5154377cfbd5e in term 0.
I20260929 17:47:46.081128 5359 raft_consensus.cc:2454] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1638062348a466a8dc5154377cfbd5e in term 0.
I20260929 17:47:46.081411 5153 leader_election.cc:302] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc93b03050c4c33b73d632503742b70, d1638062348a466a8dc5154377cfbd5e; no voters:
I20260929 17:47:46.081660 5434 raft_consensus.cc:2781] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:46.081718 5434 raft_consensus.cc:482] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:46.081759 5434 raft_consensus.cc:3037] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:46.082914 5434 raft_consensus.cc:504] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:46.083287 5434 leader_election.cc:288] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [CANDIDATE]: Term 1 election: Requested vote from peers 9bc93b03050c4c33b73d632503742b70 (127.3.183.193:40713), 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195:34813)
I20260929 17:47:46.083604 5079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309eaa8fbe054f81948d1f08a2d97b46" candidate_uuid: "d1638062348a466a8dc5154377cfbd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc93b03050c4c33b73d632503742b70"
I20260929 17:47:46.083626 5359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309eaa8fbe054f81948d1f08a2d97b46" candidate_uuid: "d1638062348a466a8dc5154377cfbd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496649c21caa4f07bdf2461ceb796d05"
I20260929 17:47:46.083726 5079 raft_consensus.cc:3037] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:46.083729 5359 raft_consensus.cc:3037] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:46.084587 5359 raft_consensus.cc:2454] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1638062348a466a8dc5154377cfbd5e in term 1.
I20260929 17:47:46.084874 5150 leader_election.cc:302] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496649c21caa4f07bdf2461ceb796d05, d1638062348a466a8dc5154377cfbd5e; no voters:
I20260929 17:47:46.084956 5079 raft_consensus.cc:2454] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1638062348a466a8dc5154377cfbd5e in term 1.
I20260929 17:47:46.085078 5434 raft_consensus.cc:2781] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:46.085263 5434 raft_consensus.cc:686] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [term 1 LEADER]: Becoming Leader. State: Replica: d1638062348a466a8dc5154377cfbd5e, State: Running, Role: LEADER
I20260929 17:47:46.085521 5434 consensus_queue.cc:237] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } }
I20260929 17:47:46.087235 5475 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: TxnStatusTablet state changed. Reason: New leader d1638062348a466a8dc5154377cfbd5e. Latest consensus state: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:46.087332 5475 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:47:46.087621 5477 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:47:46.087816 5477 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:47:46.087839 4920 catalog_manager.cc:5495] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e reported cstate change: term changed from 0 to 1, leader changed from <none> to d1638062348a466a8dc5154377cfbd5e (127.3.183.194). New cstate: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260929 17:47:46.123440 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.119161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:38567 (local address 127.0.0.1:37766)
0929 17:47:46.119323 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:46.119339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:46.119453 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:46.119753 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:46.119756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:46.119762 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:47:46.119889 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.119895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.120880 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.120883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.121512 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.121520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.122159 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.122162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.122238 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.122286 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:47:46.123133 (+ 847us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:47:46.123150 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:47:46.123227 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 17:47:46.123311 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 17:47:46.123451 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.119300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.3.183.194:38567)
0929 17:47:46.119471 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.119476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.119493 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:47:46.119573 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.119577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.119644 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.119750 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:46.119988 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.120757 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.121635 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.122034 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.122135 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.122436 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:46.123011 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:46.123133 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.123255 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.123312 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:47:46.127014 5359 raft_consensus.cc:1261] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Refusing update from remote peer d1638062348a466a8dc5154377cfbd5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:46.127082 5079 raft_consensus.cc:1261] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Refusing update from remote peer d1638062348a466a8dc5154377cfbd5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:46.127534 5475 consensus_queue.cc:1035] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:46.127712 5434 consensus_queue.cc:1035] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:46.129097 5436 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: TxnStatusTablet state changed. Reason: New leader d1638062348a466a8dc5154377cfbd5e. Latest consensus state: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:46.129225 5436 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:46.129369 5448 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: TxnStatusTablet state changed. Reason: New leader d1638062348a466a8dc5154377cfbd5e. Latest consensus state: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:46.129475 5448 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:46.132154 5436 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:46.132256 5436 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P 496649c21caa4f07bdf2461ceb796d05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:46.131681 5434 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:46.132432 5434 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:47:46.132902 5434 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:46.133009 5434 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P d1638062348a466a8dc5154377cfbd5e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:47:46.133970 5448 tablet_replica.cc:439] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1638062348a466a8dc5154377cfbd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 } } peers { permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 } } peers { permanent_uuid: "496649c21caa4f07bdf2461ceb796d05" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34813 } } }
I20260929 17:47:46.134095 5448 tablet_replica.cc:442] T 309eaa8fbe054f81948d1f08a2d97b46 P 9bc93b03050c4c33b73d632503742b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:46.134425 5482 mvcc.cc:205] Tried to move back new op lower bound from 7334723854850543616 to 7334723854688677888. Current Snapshot: MvccSnapshot[applied={T|T < 7334723854850543616}]
I20260929 17:47:46.143671 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.140073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.3.183.195:34813)
0929 17:47:46.140193 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.140197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.140393 (+ 196us) server_negotiation.cc:407] Connection header received
0929 17:47:46.140545 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.140549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.140601 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.140685 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:46.140918 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.141429 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.142328 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.142549 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.142632 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.142885 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:46.143320 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:46.143417 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.143488 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.143543 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:47:46.154598 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.151154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:60847 (local address 127.3.183.194:38567)
0929 17:47:46.151290 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.151295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.151593 (+ 298us) server_negotiation.cc:407] Connection header received
0929 17:47:46.151711 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.151714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.151758 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.151834 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:46.152054 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.152572 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.153639 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.153845 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.153928 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.154197 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:46.154264 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:46.154320 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.154409 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.154459 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:47:46.154726 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.151035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:38567 (local address 127.3.183.195:60847)
0929 17:47:46.151505 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:46.151517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:46.151615 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:46.151858 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:46.151861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:46.151868 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:47:46.151964 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.151969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.152715 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.152719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.153528 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.153538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.153943 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.153947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.154027 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.154076 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:47:46.154340 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:47:46.154346 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:47:46.154402 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:47:46.154488 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":121,"threads_started":1}
I20260929 17:47:46.161931 5219 raft_consensus.cc:1261] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Refusing update from remote peer 496649c21caa4f07bdf2461ceb796d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:46.161931 5079 raft_consensus.cc:1261] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Refusing update from remote peer 496649c21caa4f07bdf2461ceb796d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:46.162436 5460 consensus_queue.cc:1035] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1638062348a466a8dc5154377cfbd5e" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:46.162668 5436 consensus_queue.cc:1035] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc93b03050c4c33b73d632503742b70" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:46.185722 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.181598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:46061 (local address 127.3.183.195:34813)
0929 17:47:46.181710 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.181715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.181917 (+ 202us) server_negotiation.cc:407] Connection header received
0929 17:47:46.182026 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.182029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.182077 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.182154 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:46.182408 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.183171 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.184554 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.184889 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.184985 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.185281 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:46.185360 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:46.185428 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.185537 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.185587 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:47:46.185799 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.181405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34813 (local address 127.3.183.194:46061)
0929 17:47:46.181835 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:46.181848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:46.181951 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:46.182159 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:46.182161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:46.182167 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:47:46.182326 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.182332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.183257 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.183261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.184442 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:46.184452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.184988 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.184991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:46.185091 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.185134 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:47:46.185432 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:47:46.185435 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:47:46.185521 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 17:47:46.185594 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260929 17:47:46.208608 4920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:46.208797 4920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:46.211125 4920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31fbb737c0a240cda15e2a43c1f18ba6: Sending DeleteTablet for 3 replicas of tablet 64d23d2948fa422498f05a36f51c92a1
I20260929 17:47:46.211889 5199 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d23d2948fa422498f05a36f51c92a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:46 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:37762
I20260929 17:47:46.212569 5059 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d23d2948fa422498f05a36f51c92a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:46 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:38760
I20260929 17:47:46.213033 5339 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d23d2948fa422498f05a36f51c92a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:46 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:58446
I20260929 17:47:46.213076 5503 tablet_replica.cc:330] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: stopping tablet replica
I20260929 17:47:46.213328 5503 raft_consensus.cc:2229] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:46.213521 5503 raft_consensus.cc:2258] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:46.214269 5503 ts_tablet_manager.cc:1905] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:46.217720 5503 ts_tablet_manager.cc:1918] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:46.217806 5503 log.cc:1196] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/64d23d2948fa422498f05a36f51c92a1
I20260929 17:47:46.217935 5504 tablet_replica.cc:330] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: stopping tablet replica
I20260929 17:47:46.218079 5503 ts_tablet_manager.cc:1939] T 64d23d2948fa422498f05a36f51c92a1 P 9bc93b03050c4c33b73d632503742b70: Deleting consensus metadata
I20260929 17:47:46.218251 5504 raft_consensus.cc:2229] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:47:46.218709 5504 raft_consensus.cc:2258] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:46.218847 4909 catalog_manager.cc:4841] TS 9bc93b03050c4c33b73d632503742b70 (127.3.183.193:40713): tablet 64d23d2948fa422498f05a36f51c92a1 (table test-table [id=b92f3ae0b5bf4094ad50bbeff571154d]) successfully deleted
I20260929 17:47:46.219172 5501 tablet_replica.cc:330] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: stopping tablet replica
I20260929 17:47:46.219314 5501 raft_consensus.cc:2229] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:46.219441 5501 raft_consensus.cc:2258] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:46.219831 5504 ts_tablet_manager.cc:1905] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:46.220158 5501 ts_tablet_manager.cc:1905] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:46.223207 5501 ts_tablet_manager.cc:1918] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:46.223290 5501 log.cc:1196] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/64d23d2948fa422498f05a36f51c92a1
I20260929 17:47:46.223587 5504 ts_tablet_manager.cc:1918] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:46.223701 5504 log.cc:1196] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/64d23d2948fa422498f05a36f51c92a1
I20260929 17:47:46.223759 5501 ts_tablet_manager.cc:1939] T 64d23d2948fa422498f05a36f51c92a1 P d1638062348a466a8dc5154377cfbd5e: Deleting consensus metadata
I20260929 17:47:46.224305 5504 ts_tablet_manager.cc:1939] T 64d23d2948fa422498f05a36f51c92a1 P 496649c21caa4f07bdf2461ceb796d05: Deleting consensus metadata
I20260929 17:47:46.224530 4909 catalog_manager.cc:4841] TS d1638062348a466a8dc5154377cfbd5e (127.3.183.194:38567): tablet 64d23d2948fa422498f05a36f51c92a1 (table test-table [id=b92f3ae0b5bf4094ad50bbeff571154d]) successfully deleted
I20260929 17:47:46.225044 4906 catalog_manager.cc:4841] TS 496649c21caa4f07bdf2461ceb796d05 (127.3.183.195:34813): tablet 64d23d2948fa422498f05a36f51c92a1 (table test-table [id=b92f3ae0b5bf4094ad50bbeff571154d]) successfully deleted
Sep 29 17:47:46 dist-test-slave-wbpt krb5kdc[4877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:47:46.237712 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.225134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.3.183.193:40713)
0929 17:47:46.225253 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.225258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.225812 (+ 554us) server_negotiation.cc:407] Connection header received
0929 17:47:46.225920 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.225924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.225979 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.226080 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:46.226864 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.227659 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.228961 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.229345 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.229458 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.233463 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:46.233478 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:46.233494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:46.233523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:46.235618 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:46.236191 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:46.236198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:46.236200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:46.236329 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:46.236620 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:46.236623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:46.236625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:46.236938 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:46.237184 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.237412 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.237543 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 17:47:46.238279 5109 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38772
I20260929 17:47:46.249627 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:46.240047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.3.183.254:38313)
0929 17:47:46.240182 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:46.240187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:46.240334 (+ 147us) server_negotiation.cc:407] Connection header received
0929 17:47:46.240436 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:46.240439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:46.240495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:46.240584 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:46.241473 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.242312 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.243586 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:46.244082 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:46.244188 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:46.245844 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:46.245848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:46.245851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:46.245879 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:46.247555 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:46.248069 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:46.248073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:46.248075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:46.248159 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:46.248453 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:46.248457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:46.248459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:46.248702 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:46.248807 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:46.249306 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 17:47:46.249454 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 17:47:46.250247 4920 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55038
I20260929 17:47:46.251662 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4994
I20260929 17:47:46.260089 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 5134
I20260929 17:47:46.266424 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 5274
I20260929 17:47:46.271430 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 4890
2026-09-29T17:47:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2907 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:46 dist-test-slave-wbpt krb5kdc[5510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:46 dist-test-slave-wbpt krb5kdc[5510](info): set up 2 sockets
Sep 29 17:47:46 dist-test-slave-wbpt krb5kdc[5510](info): commencing operation
krb5kdc: starting...
W20260929 17:47:47.235311 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:47 dist-test-slave-wbpt krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:47.349141 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34143
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:34143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:47.453056 5523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:47.453296 5523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:47.453332 5523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:47.457268 5523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:47.457315 5523 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:47:47.457331 5523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:47.457348 5523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:47.457362 5523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:47.457381 5523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:47.462379 5523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:34143
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:47.463303 5523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:47.464262 5523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:47.472007 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:47.472079 5531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:47.472003 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:47.472369 5523 server_base.cc:1034] running on GCE node
I20260929 17:47:47.472870 5523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:47.473966 5523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:47.475121 5523 hybrid_clock.cc:648] HybridClock initialized: now 1790704067475113 us; error 34 us; skew 500 ppm
Sep 29 17:47:47 dist-test-slave-wbpt krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:47.477691 5523 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:47.478488 5523 webserver.cc:460] Webserver started at http://127.3.183.254:36241/ using document root <none> and password file <none>
I20260929 17:47:47.478994 5523 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:47.479063 5523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:47.479238 5523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:47.481206 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "bb84761044a343c39e579c3965709f6d"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "f374f541e829e88db6d11c0cf5650c2d"
server_key_iv: "c376b325e720d3d1dfe86e036c400c7f"
server_key_version: "encryptionkey@0"
I20260929 17:47:47.481741 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "bb84761044a343c39e579c3965709f6d"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "f374f541e829e88db6d11c0cf5650c2d"
server_key_iv: "c376b325e720d3d1dfe86e036c400c7f"
server_key_version: "encryptionkey@0"
I20260929 17:47:47.485219 5523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:47:47.487422 5538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:47.488591 5523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:47.488709 5523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "bb84761044a343c39e579c3965709f6d"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "f374f541e829e88db6d11c0cf5650c2d"
server_key_iv: "c376b325e720d3d1dfe86e036c400c7f"
server_key_version: "encryptionkey@0"
I20260929 17:47:47.488798 5523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:47.497694 5523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:47.500520 5523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:47.500694 5523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:47.507650 5523 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:34143
I20260929 17:47:47.507701 5600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:34143 every 8 connection(s)
I20260929 17:47:47.508535 5523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:47.511229 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:47.514330 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 5523
I20260929 17:47:47.514416 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:47.514645 3807 external_mini_cluster.cc:1428] Setting key d95edf6bc203c2a79cfb3626df4f2607
I20260929 17:47:47.517115 5601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Bootstrap starting.
I20260929 17:47:47.519608 5601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:47.520515 5601 log.cc:824] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:47.522332 5601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: No bootstrap required, opened a new log
Sep 29 17:47:47 dist-test-slave-wbpt krb5kdc[5510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:47.525661 5601 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84761044a343c39e579c3965709f6d" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34143 } }
I20260929 17:47:47.525913 5601 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:47.525974 5601 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb84761044a343c39e579c3965709f6d, State: Initialized, Role: FOLLOWER
I20260929 17:47:47.526371 5601 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84761044a343c39e579c3965709f6d" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34143 } }
I20260929 17:47:47.526492 5601 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:47.526556 5601 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:47.526648 5601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:47.527611 5601 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84761044a343c39e579c3965709f6d" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34143 } }
I20260929 17:47:47.527956 5601 leader_election.cc:302] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb84761044a343c39e579c3965709f6d; no voters:
I20260929 17:47:47.528187 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:47.515860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.3.183.254:34143)
0929 17:47:47.516229 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:47.516245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:47.516279 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:47:47.516863 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:47.516879 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:47.517203 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:47.517486 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:47.518393 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.519362 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.520549 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.521028 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.521143 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:47.523709 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:47.523714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:47.523726 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:47.523753 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:47.525326 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:47.525807 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:47.525811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:47.525815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:47.525923 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:47.526240 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:47.526243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:47.526245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:47.526713 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:47.526882 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:47.527125 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 17:47:47.527385 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260929 17:47:47.528230 5601 leader_election.cc:288] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:47.528317 5606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:47.528496 5606 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [term 1 LEADER]: Becoming Leader. State: Replica: bb84761044a343c39e579c3965709f6d, State: Running, Role: LEADER
I20260929 17:47:47.528769 5606 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84761044a343c39e579c3965709f6d" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34143 } }
I20260929 17:47:47.529331 5601 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:47.529874 5608 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb84761044a343c39e579c3965709f6d. Latest consensus state: current_term: 1 leader_uuid: "bb84761044a343c39e579c3965709f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84761044a343c39e579c3965709f6d" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34143 } } }
I20260929 17:47:47.529901 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb84761044a343c39e579c3965709f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84761044a343c39e579c3965709f6d" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34143 } } }
I20260929 17:47:47.529992 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:47.529992 5608 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:47.530634 5612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:47.533624 5612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:47.539153 5612 catalog_manager.cc:1347] Generated new cluster ID: 90693931ce554480860397a497f990e3
I20260929 17:47:47.539223 5612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:47.547269 5612 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:47.548261 5612 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:47.560235 5612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Generated new TSK 0
I20260929 17:47:47.560895 5612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:47:47.595106 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34143
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:47.737740 5627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:47.738008 5627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:47.738055 5627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:47.742033 5627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:47.742087 5627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:47.742126 5627 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:47:47.742138 5627 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:47:47.742182 5627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:47:47.747488 5627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:34143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:47.748580 5627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:47.749617 5627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:47.758883 5632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:47.758921 5635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:47.759047 5627 server_base.cc:1034] running on GCE node
W20260929 17:47:47.758932 5633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:47.759531 5627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:47.760141 5627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:47.761305 5627 hybrid_clock.cc:648] HybridClock initialized: now 1790704067761279 us; error 43 us; skew 500 ppm
Sep 29 17:47:47 dist-test-slave-wbpt krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:47.764006 5627 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:47:47.764846 5627 webserver.cc:460] Webserver started at http://127.3.183.193:44617/ using document root <none> and password file <none>
I20260929 17:47:47.765367 5627 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:47.765439 5627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:47.765635 5627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:47.767609 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "7c4c3649994f4233aca644a8fb236435"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "03601b88fe6d3a0318be7dc79d5b7427"
server_key_iv: "bfab70456daeadbdf2bb1e70f2ddea81"
server_key_version: "encryptionkey@0"
I20260929 17:47:47.768168 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "7c4c3649994f4233aca644a8fb236435"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "03601b88fe6d3a0318be7dc79d5b7427"
server_key_iv: "bfab70456daeadbdf2bb1e70f2ddea81"
server_key_version: "encryptionkey@0"
I20260929 17:47:47.771914 5627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:47:47.774317 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:47.775576 5627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:47.775691 5627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "7c4c3649994f4233aca644a8fb236435"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "03601b88fe6d3a0318be7dc79d5b7427"
server_key_iv: "bfab70456daeadbdf2bb1e70f2ddea81"
server_key_version: "encryptionkey@0"
I20260929 17:47:47.775839 5627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:47.801474 5627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:47.804632 5627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:47.804811 5627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:47.806237 5627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:47.806288 5627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:47.806339 5627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:47.806367 5627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:47:47 dist-test-slave-wbpt krb5kdc[5510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:47.815901 5627 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:34767
I20260929 17:47:47.816038 5757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:34767 every 8 connection(s)
I20260929 17:47:47.816886 5627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:47:47.819103 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:47.807119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:53527 (local address 127.3.183.254:34143)
0929 17:47:47.807262 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:47.807268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:47.808162 (+ 894us) server_negotiation.cc:407] Connection header received
0929 17:47:47.809249 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:47.809252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:47.809307 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:47.809378 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:47.810833 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.811401 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.812735 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.812990 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.813082 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:47.815427 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:47.815430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:47.815431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:47.815457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:47.817058 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:47.817628 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:47.817632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:47.817634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:47.817724 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:47.818054 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:47.818056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:47.818058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:47.818238 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:47.818314 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:47.818864 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 17:47:47.818973 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:47:47.819968 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:47.807512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34143 (local address 127.3.183.193:53527)
0929 17:47:47.808002 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:47.808050 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:47.809022 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:47.809590 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:47.809600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:47.810013 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:47.810695 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:47.810715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.811519 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.811524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:47.812609 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:47.812627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.813131 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.813136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:47.813307 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:47.813842 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:47.813858 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:47.815264 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:47.817159 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:47.817178 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:47.817192 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:47.817501 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:47.817819 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:47.817821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:47.817823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:47.817967 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:47.818368 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:47.818374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:47.818630 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 17:47:47.818799 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:47:47.819061 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260929 17:47:47.821256 5758 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34143
I20260929 17:47:47.821393 5758 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:47.821858 5758 heartbeater.cc:511] Master 127.3.183.254:34143 requested a full tablet report, sending...
I20260929 17:47:47.822516 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 5627
I20260929 17:47:47.822623 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:47:47.822918 3807 external_mini_cluster.cc:1428] Setting key 294a31a2d4471029329457edb7715e0d
I20260929 17:47:47.823288 5555 ts_manager.cc:194] Registered new tserver with Master: 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767)
I20260929 17:47:47.824803 5555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:53527
I20260929 17:47:47.831081 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:47.823816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:52597 (local address 127.3.183.254:34143)
0929 17:47:47.823927 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:47.823931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:47.823943 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:47:47.823976 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:47.823979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:47.824017 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:47.824095 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:47.824575 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.825332 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.826604 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.826925 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.826997 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:47.828208 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:47.828212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:47.828215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:47.828239 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:47.829546 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:47.829930 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:47.829933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:47.829935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:47.830012 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:47.830316 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:47.830318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:47.830319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:47.830510 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:47.830591 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:47.830859 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 17:47:47.830958 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":51}
I20260929 17:47:47.831166 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:47.823487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34143 (local address 127.3.183.193:52597)
0929 17:47:47.823761 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:47.823797 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:47.823877 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:47.824120 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:47.824123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:47.824339 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:47.824465 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:47.824473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.825633 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.825636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:47.826470 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:47.826481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:47.827150 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:47.827153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:47.827238 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:47.827722 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:47.827733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:47.828076 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:47.829645 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:47.829648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:47.829651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:47.829843 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:47.830118 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:47.830122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:47.830124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:47.830229 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:47.830601 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:47.830605 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:47.830712 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 17:47:47.830818 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:47:47.830979 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:47:47.850553 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34143
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:47.960652 5767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:47.961005 5767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:47.961083 5767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:47.966288 5767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:47.966360 5767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:47.966423 5767 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:47:47.966449 5767 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:47:47.966506 5767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:47:47.973381 5767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:34143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:47.974609 5767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:47.975690 5767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:47.987963 5773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:47.987963 5772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:47.988309 5767 server_base.cc:1034] running on GCE node
W20260929 17:47:47.988442 5775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:47.988910 5767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:47.989753 5767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:47.990952 5767 hybrid_clock.cc:648] HybridClock initialized: now 1790704067990924 us; error 33 us; skew 500 ppm
Sep 29 17:47:47 dist-test-slave-wbpt krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:48.000274 5767 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:47:48.001143 5767 webserver.cc:460] Webserver started at http://127.3.183.194:44559/ using document root <none> and password file <none>
I20260929 17:47:48.001711 5767 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:48.001780 5767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:48.001981 5767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:48.004104 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "058c1156cfd6467bbe2f320c0a9e9aea"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "ac3e9e289dc42b3508a3c4bb114d3935"
server_key_iv: "909118fc2ba8406c5567156b8cc87110"
server_key_version: "encryptionkey@0"
I20260929 17:47:48.004669 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "058c1156cfd6467bbe2f320c0a9e9aea"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "ac3e9e289dc42b3508a3c4bb114d3935"
server_key_iv: "909118fc2ba8406c5567156b8cc87110"
server_key_version: "encryptionkey@0"
I20260929 17:47:48.008330 5767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:47:48.010735 5782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:48.012004 5767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:48.012109 5767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "058c1156cfd6467bbe2f320c0a9e9aea"
format_stamp: "Formatted at 2026-09-29 17:47:47 on dist-test-slave-wbpt"
server_key: "ac3e9e289dc42b3508a3c4bb114d3935"
server_key_iv: "909118fc2ba8406c5567156b8cc87110"
server_key_version: "encryptionkey@0"
I20260929 17:47:48.012214 5767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:48.028960 5767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:48.032164 5767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:48.032331 5767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:48.033798 5767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:48.033849 5767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:48.033900 5767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:48.033927 5767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:48.042922 5767 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:40543
I20260929 17:47:48.042999 5897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:40543 every 8 connection(s)
Sep 29 17:47:48 dist-test-slave-wbpt krb5kdc[5510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:48.043833 5767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:47:48.046758 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 5767
I20260929 17:47:48.046867 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:47:48.047137 3807 external_mini_cluster.cc:1428] Setting key 8614b402b7ee011f2289ee913b67131f
I20260929 17:47:48.047222 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.034815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:33587 (local address 127.3.183.254:34143)
0929 17:47:48.034953 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.034958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.035840 (+ 882us) server_negotiation.cc:407] Connection header received
0929 17:47:48.037127 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.037131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.037205 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.037276 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:48.038742 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.039572 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.040989 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.041331 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.041447 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.044005 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:48.044008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:48.044010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:48.044034 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:48.045425 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.045860 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.045862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.045863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.045935 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.046200 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.046202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.046203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.046361 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:48.046426 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.047001 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.047099 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:47:48.048081 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.035230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34143 (local address 127.3.183.194:33587)
0929 17:47:48.035659 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.035709 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.036903 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.037465 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.037473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.037924 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:48.038569 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.038588 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.039706 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.039710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.040865 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.040874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.041480 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.041484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.041681 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.042273 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:48.042288 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:48.043819 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:48.045513 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.045517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.045526 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.045766 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.045996 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.045998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.045999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.046130 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.046428 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:48.046432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:48.046722 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.046920 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.047225 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20260929 17:47:48.049257 5898 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34143
I20260929 17:47:48.049397 5898 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:48.049927 5898 heartbeater.cc:511] Master 127.3.183.254:34143 requested a full tablet report, sending...
I20260929 17:47:48.050943 5554 ts_manager.cc:194] Registered new tserver with Master: 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194:40543)
I20260929 17:47:48.051586 5554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:33587
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:47:48.059048 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.051551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34143 (local address 127.3.183.194:41097)
0929 17:47:48.051977 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.051989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.052082 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.052304 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.052307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.052531 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:48.052680 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.052688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.053765 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.053768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.054659 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.054670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.055101 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.055105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.055208 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.055676 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:48.055690 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:48.056113 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:48.057698 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.057701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.057702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.057902 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.058113 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.058115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.058116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.058188 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.058522 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:48.058523 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:48.058594 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.058700 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.058843 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260929 17:47:48.059039 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.051527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:41097 (local address 127.3.183.254:34143)
0929 17:47:48.051645 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.051649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.052061 (+ 412us) server_negotiation.cc:407] Connection header received
0929 17:47:48.052173 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.052175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.052213 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.052268 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:48.052818 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.053594 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.054780 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.054989 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.055068 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.056238 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:48.056241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:48.056243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:48.056265 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:48.057573 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.057976 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.057978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.057979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.058050 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.058240 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.058242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.058243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.058450 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:48.058528 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.058788 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.058894 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:47:48.074084 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34143
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:48.182119 5907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:48.182392 5907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:48.182440 5907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:48.186391 5907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:48.186447 5907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:48.186486 5907 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:47:48.186498 5907 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:47:48.186528 5907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:47:48.191694 5907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:34143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:48.192746 5907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:48.193691 5907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:48.203259 5912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:48.203339 5907 server_base.cc:1034] running on GCE node
W20260929 17:47:48.203279 5915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:48.203302 5913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:48.203931 5907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:48.204566 5907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:48.205718 5907 hybrid_clock.cc:648] HybridClock initialized: now 1790704068205698 us; error 31 us; skew 500 ppm
Sep 29 17:47:48 dist-test-slave-wbpt krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704068, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:48.208287 5907 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:47:48.209129 5907 webserver.cc:460] Webserver started at http://127.3.183.195:39015/ using document root <none> and password file <none>
I20260929 17:47:48.209648 5907 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:48.209712 5907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:48.209892 5907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:48.211880 5907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "252100efde454c8b87a20e9177c23724"
format_stamp: "Formatted at 2026-09-29 17:47:48 on dist-test-slave-wbpt"
server_key: "d727919a08f4cae057ad35c6bed926bf"
server_key_iv: "df7c1a6e2c9dc8dd47e94009baeab6e5"
server_key_version: "encryptionkey@0"
I20260929 17:47:48.212402 5907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "252100efde454c8b87a20e9177c23724"
format_stamp: "Formatted at 2026-09-29 17:47:48 on dist-test-slave-wbpt"
server_key: "d727919a08f4cae057ad35c6bed926bf"
server_key_iv: "df7c1a6e2c9dc8dd47e94009baeab6e5"
server_key_version: "encryptionkey@0"
I20260929 17:47:48.215968 5907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:47:48.218312 5922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:48.219511 5907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:48.219619 5907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "252100efde454c8b87a20e9177c23724"
format_stamp: "Formatted at 2026-09-29 17:47:48 on dist-test-slave-wbpt"
server_key: "d727919a08f4cae057ad35c6bed926bf"
server_key_iv: "df7c1a6e2c9dc8dd47e94009baeab6e5"
server_key_version: "encryptionkey@0"
I20260929 17:47:48.219705 5907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:48.240139 5907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:48.242759 5907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:48.242925 5907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:48.244372 5907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:48.244421 5907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:48.244472 5907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:48.244508 5907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:47:48 dist-test-slave-wbpt krb5kdc[5510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704068, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:48.254994 6037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:40131 every 8 connection(s)
I20260929 17:47:48.255225 5907 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:40131
I20260929 17:47:48.256239 5907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:47:48.258438 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.245181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:55733 (local address 127.3.183.254:34143)
0929 17:47:48.245301 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.245305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.246317 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 17:47:48.247493 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.247497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.247562 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.247629 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:48.249175 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.249950 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.251197 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.251541 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.251650 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.254963 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:48.254967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:48.254970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:48.255002 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:48.256525 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.256996 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.257000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.257002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.257081 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.257420 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.257423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.257425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.257609 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:48.257701 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.258162 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.258283 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:47:48.259234 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.245667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34143 (local address 127.3.183.195:55733)
0929 17:47:48.246165 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.246207 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.247178 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.248067 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.248074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.248467 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:48.249011 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.249026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.250070 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.250073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.251074 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.251084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.251687 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.251691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.251913 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.252569 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:48.252592 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:48.254741 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:48.256633 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.256640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.256655 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.256902 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.257178 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.257181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.257183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.257335 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.257714 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:48.257720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:48.257949 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.258106 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.258429 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260929 17:47:48.260187 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 5907
I20260929 17:47:48.260288 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:47:48.260546 3807 external_mini_cluster.cc:1428] Setting key fd0dbbb022dee0ca7d871fec94f30c95
I20260929 17:47:48.260605 6038 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34143
I20260929 17:47:48.260735 6038 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:48.261241 6038 heartbeater.cc:511] Master 127.3.183.254:34143 requested a full tablet report, sending...
I20260929 17:47:48.262302 5555 ts_manager.cc:194] Registered new tserver with Master: 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131)
I20260929 17:47:48.262900 5555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:55733
I20260929 17:47:48.269001 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.262963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:49023 (local address 127.3.183.254:34143)
0929 17:47:48.263062 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.263066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.263268 (+ 202us) server_negotiation.cc:407] Connection header received
0929 17:47:48.263342 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.263345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.263390 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.263456 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:48.263917 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.264391 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.265265 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.265471 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.265543 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.266491 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:48.266494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:48.266496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:48.266516 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:48.267739 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.268133 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.268135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.268136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.268203 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.268400 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.268402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.268403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.268557 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:48.268640 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.268815 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.268898 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 17:47:48.269001 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.262877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34143 (local address 127.3.183.195:49023)
0929 17:47:48.263189 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.263203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.263284 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.263486 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.263489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.263682 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:48.263823 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.263830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.264504 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.264507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.265164 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.265173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.265572 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.265575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.265645 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.266048 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:48.266060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:48.266368 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:48.267865 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.267869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.267871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.268059 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.268271 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.268273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.268275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.268340 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.268621 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:48.268623 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:48.268693 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.268765 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.268867 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
I20260929 17:47:48.274504 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:47:48 dist-test-slave-wbpt krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:47:48 dist-test-slave-wbpt krb5kdc[5510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:48.298542 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.290712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.3.183.254:34143)
0929 17:47:48.290846 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.290849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.290861 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:47:48.290899 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.290901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.290940 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.291022 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:48.291721 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.292303 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.293178 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.293411 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.293517 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.295818 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:48.295822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:48.295826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:48.295852 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:48.297147 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.297497 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.297500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.297502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.297580 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.297776 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.297778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.297779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.298005 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:48.298106 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.298300 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.298386 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:47:48.301146 5555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:47:48.304078 5555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:48.319185 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.311874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:40131 (local address 127.0.0.1:34244)
0929 17:47:48.312355 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.312400 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.312517 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.314371 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.314377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.314401 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:48.314557 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.314565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.316037 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.316040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.317533 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.317545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.318627 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.318631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.318751 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.318800 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.318888 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.319025 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":40,"thread_start_us":72,"threads_started":1}
I20260929 17:47:48.319813 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.312900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40543 (local address 127.0.0.1:34706)
0929 17:47:48.313352 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.313365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.313513 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.314439 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.314441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.314454 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:48.314677 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.314682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.316037 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.316139 (+ 102us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.318118 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.318128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.319390 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.319393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.319490 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.319514 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.319580 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.319649 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":84,"thread_start_us":82,"threads_started":1}
I20260929 17:47:48.320053 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.313401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.3.183.195:40131)
0929 17:47:48.313953 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.313964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.313987 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:47:48.314057 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.314062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.314220 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.314559 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:48.314685 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.315893 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.317679 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.318500 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.319560 (+ 1060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.319701 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.319750 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.319887 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":91,"threads_started":1}
I20260929 17:47:48.320118 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.312543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.3.183.194:40543)
0929 17:47:48.312881 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.312894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.313442 (+ 548us) server_negotiation.cc:407] Connection header received
0929 17:47:48.313656 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.313662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.314147 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.314277 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:48.314810 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.315919 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.318250 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.319054 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.319133 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.319274 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.319633 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.319738 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260929 17:47:48.320122 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.313148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:34767 (local address 127.0.0.1:45510)
0929 17:47:48.315323 (+ 2175us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.315336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.315427 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.316104 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.316107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.316135 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:48.316244 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.316250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.317602 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.317605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.318672 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.318682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.319705 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.319708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.319825 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.319850 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.319921 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.319988 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2115,"thread_start_us":90,"threads_started":1}
I20260929 17:47:48.320600 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.313348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.3.183.193:34767)
0929 17:47:48.313863 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.313879 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.315522 (+ 1643us) server_negotiation.cc:407] Connection header received
0929 17:47:48.315601 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.315605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.315810 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.315961 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:48.316378 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.317472 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.318795 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.319413 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.319505 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.319648 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.320374 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.320474 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":114,"threads_started":1}
I20260929 17:47:48.322979 5832 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf7a35348224ae2a2cac8b9da28440e (DEFAULT_TABLE table=test-table [id=cde91ad93adf4ebcbcadff2ff0fd57fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:48.323040 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf7a35348224ae2a2cac8b9da28440e (DEFAULT_TABLE table=test-table [id=cde91ad93adf4ebcbcadff2ff0fd57fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:48.323167 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf7a35348224ae2a2cac8b9da28440e (DEFAULT_TABLE table=test-table [id=cde91ad93adf4ebcbcadff2ff0fd57fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:48.324040 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf7a35348224ae2a2cac8b9da28440e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.324043 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf7a35348224ae2a2cac8b9da28440e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.324043 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf7a35348224ae2a2cac8b9da28440e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.331007 6061 tablet_bootstrap.cc:492] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Bootstrap starting.
I20260929 17:47:48.331687 6062 tablet_bootstrap.cc:492] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Bootstrap starting.
I20260929 17:47:48.331670 6063 tablet_bootstrap.cc:492] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Bootstrap starting.
I20260929 17:47:48.333052 6061 tablet_bootstrap.cc:654] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.333709 6063 tablet_bootstrap.cc:654] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.333738 6061 log.cc:824] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:48.333765 6062 tablet_bootstrap.cc:654] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.334532 6062 log.cc:824] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:48.334627 6063 log.cc:824] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:48.335556 6061 tablet_bootstrap.cc:492] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: No bootstrap required, opened a new log
I20260929 17:47:48.335743 6061 ts_tablet_manager.cc:1397] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:47:48.336262 6063 tablet_bootstrap.cc:492] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: No bootstrap required, opened a new log
I20260929 17:47:48.336313 6062 tablet_bootstrap.cc:492] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: No bootstrap required, opened a new log
I20260929 17:47:48.336417 6063 ts_tablet_manager.cc:1397] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 17:47:48.336469 6062 ts_tablet_manager.cc:1397] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:47:48.339031 6063 raft_consensus.cc:348] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.339214 6063 raft_consensus.cc:374] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.339250 6063 raft_consensus.cc:729] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252100efde454c8b87a20e9177c23724, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.339627 6063 consensus_queue.cc:260] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.339550 6061 raft_consensus.cc:348] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.339838 6061 raft_consensus.cc:374] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.339898 6061 raft_consensus.cc:729] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.340283 6038 heartbeater.cc:503] Master 127.3.183.254:34143 was elected leader, sending a full tablet report...
I20260929 17:47:48.340334 6061 consensus_queue.cc:260] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.340391 6062 raft_consensus.cc:348] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.340591 6063 ts_tablet_manager.cc:1428] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 17:47:48.340607 6062 raft_consensus.cc:374] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.340657 6062 raft_consensus.cc:729] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c3649994f4233aca644a8fb236435, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.341037 6062 consensus_queue.cc:260] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.341107 5898 heartbeater.cc:503] Master 127.3.183.254:34143 was elected leader, sending a full tablet report...
I20260929 17:47:48.341269 6061 ts_tablet_manager.cc:1428] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260929 17:47:48.341845 6062 ts_tablet_manager.cc:1428] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:47:48.342139 5758 heartbeater.cc:503] Master 127.3.183.254:34143 was elected leader, sending a full tablet report...
I20260929 17:47:48.486901 6068 raft_consensus.cc:482] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:48.487119 6068 raft_consensus.cc:504] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.488350 6068 leader_election.cc:288] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131), 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767)
I20260929 17:47:48.492476 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.488652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:40131 (local address 127.3.183.194:33281)
0929 17:47:48.488773 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.488789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.488910 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.489182 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.489185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.489204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:48.489336 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.489344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.490276 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.490279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.491187 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.491197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.492073 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.492076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.492181 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.492203 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.492269 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.492342 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 17:47:48.492564 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.488744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:33281 (local address 127.3.183.195:40131)
0929 17:47:48.488864 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.488869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.488886 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:47:48.489013 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.489017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.489067 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.489161 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:48.489515 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.490158 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.491317 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.491943 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.492045 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.492097 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.492308 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.492407 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:47:48.493294 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.488652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:34767 (local address 127.3.183.194:34545)
0929 17:47:48.489197 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.489210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.489302 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.489558 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.489561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.489580 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:48.489687 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.489693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.490483 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.490485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.491874 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.491885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.492934 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.492938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.493034 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.493057 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.493132 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.493184 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":114,"thread_start_us":186,"threads_started":1}
I20260929 17:47:48.493351 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.488792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:34545 (local address 127.3.183.193:34767)
0929 17:47:48.488938 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.488943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.489293 (+ 350us) server_negotiation.cc:407] Connection header received
0929 17:47:48.489381 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.489385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.489437 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.489514 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:48.489804 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.490365 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.492077 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.492787 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.492930 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.493016 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.493156 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.493233 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:47:48.493358 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf7a35348224ae2a2cac8b9da28440e" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724" is_pre_election: true
I20260929 17:47:48.493660 5990 raft_consensus.cc:2454] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:48.493989 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf7a35348224ae2a2cac8b9da28440e" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435" is_pre_election: true
I20260929 17:47:48.494145 5786 leader_election.cc:302] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724; no voters:
I20260929 17:47:48.494261 5712 raft_consensus.cc:2454] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:48.494410 6068 raft_consensus.cc:2781] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:48.494544 6068 raft_consensus.cc:482] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:48.494601 6068 raft_consensus.cc:3037] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:48.495591 6068 raft_consensus.cc:504] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.495911 6068 leader_election.cc:288] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Requested vote from peers 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131), 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767)
I20260929 17:47:48.496225 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf7a35348224ae2a2cac8b9da28440e" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435"
I20260929 17:47:48.496357 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf7a35348224ae2a2cac8b9da28440e" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724"
I20260929 17:47:48.496356 5712 raft_consensus.cc:3037] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:48.496474 5990 raft_consensus.cc:3037] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:48.497758 5990 raft_consensus.cc:2454] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:48.497758 5712 raft_consensus.cc:2454] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:48.498070 5785 leader_election.cc:302] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 7c4c3649994f4233aca644a8fb236435; no voters:
I20260929 17:47:48.498303 6068 raft_consensus.cc:2781] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:48.498543 6068 raft_consensus.cc:686] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 LEADER]: Becoming Leader. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Running, Role: LEADER
I20260929 17:47:48.498871 6068 consensus_queue.cc:237] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } }
I20260929 17:47:48.501956 5555 catalog_manager.cc:5495] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea reported cstate change: term changed from 0 to 1, leader changed from <none> to 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194). New cstate: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } health_report { overall_health: HEALTHY } } }
W20260929 17:47:48.507599 6039 tablet.cc:2369] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:47:48.544939 5899 tablet.cc:2369] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:47:48.568246 5759 tablet.cc:2369] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 17:47:48 dist-test-slave-wbpt krb5kdc[5510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:48.581640 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.573400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34143 (local address 127.0.0.1:55008)
0929 17:47:48.573693 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.573705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.573809 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.574024 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.574026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.574419 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:48.574521 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.574528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.575296 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.575299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.575918 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.575926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.576317 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.576320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.576381 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.576926 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:48.576938 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:48.578304 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:48.579943 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.579953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.579958 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.580279 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.580604 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:48.580607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:48.580608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:48.580691 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:48.581154 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:48.581160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:48.581269 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.581367 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.581493 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260929 17:47:48.581730 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.573498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.3.183.254:34143)
0929 17:47:48.573651 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.573655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.573776 (+ 121us) server_negotiation.cc:407] Connection header received
0929 17:47:48.573883 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.573885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.573925 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.573992 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:48.574670 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.575189 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.576020 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.576225 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.576298 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.578422 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:48.578425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:48.578426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:48.578447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:48.579839 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.580397 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.580400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.580401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.580514 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:48.580800 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:48.580803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:48.580804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:48.581021 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:48.581114 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.581457 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.581584 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 17:47:48.585485 5555 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:55008:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:47:48.586210 5555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:48.590919 5832 tablet_service.cc:1467] Processing CreateTablet for tablet cf4d97c139e54d00891518e67697c190 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de0e359886d34660a1665e1fda61e9fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:47:48.590870 5692 tablet_service.cc:1467] Processing CreateTablet for tablet cf4d97c139e54d00891518e67697c190 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de0e359886d34660a1665e1fda61e9fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:47:48.591084 5972 tablet_service.cc:1467] Processing CreateTablet for tablet cf4d97c139e54d00891518e67697c190 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de0e359886d34660a1665e1fda61e9fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:47:48.591636 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4d97c139e54d00891518e67697c190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.591636 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4d97c139e54d00891518e67697c190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.591758 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4d97c139e54d00891518e67697c190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.594470 6063 tablet_bootstrap.cc:492] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: Bootstrap starting.
I20260929 17:47:48.594511 6061 tablet_bootstrap.cc:492] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: Bootstrap starting.
I20260929 17:47:48.594803 6062 tablet_bootstrap.cc:492] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: Bootstrap starting.
I20260929 17:47:48.595449 6061 tablet_bootstrap.cc:654] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.595449 6063 tablet_bootstrap.cc:654] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.595638 6062 tablet_bootstrap.cc:654] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.596695 6061 tablet_bootstrap.cc:492] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: No bootstrap required, opened a new log
I20260929 17:47:48.596781 6061 ts_tablet_manager.cc:1397] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 17:47:48.597071 6062 tablet_bootstrap.cc:492] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: No bootstrap required, opened a new log
I20260929 17:47:48.597157 6062 ts_tablet_manager.cc:1397] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 17:47:48.597187 6061 raft_consensus.cc:348] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.597299 6061 raft_consensus.cc:374] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.597333 6061 raft_consensus.cc:729] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.597452 6061 consensus_queue.cc:260] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.597608 6062 raft_consensus.cc:348] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.597697 6061 ts_tablet_manager.cc:1428] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:47:48.597702 6062 raft_consensus.cc:374] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.597671 6068 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.597736 6062 raft_consensus.cc:729] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c3649994f4233aca644a8fb236435, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.597802 6068 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:48.597848 6062 consensus_queue.cc:260] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.598080 6062 ts_tablet_manager.cc:1428] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:47:48.598080 6069 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.598637 6069 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:48.598778 6063 tablet_bootstrap.cc:492] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: No bootstrap required, opened a new log
I20260929 17:47:48.598877 6063 ts_tablet_manager.cc:1397] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 17:47:48.599397 6063 raft_consensus.cc:348] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.599502 6063 raft_consensus.cc:374] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.599524 6063 raft_consensus.cc:729] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252100efde454c8b87a20e9177c23724, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.599633 6063 consensus_queue.cc:260] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.599926 6063 ts_tablet_manager.cc:1428] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:47:48.599968 6067 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.600131 6067 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:48.805297 6068 raft_consensus.cc:482] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:48.805431 6068 raft_consensus.cc:504] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.805851 6068 leader_election.cc:288] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131)
I20260929 17:47:48.806344 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4d97c139e54d00891518e67697c190" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724" is_pre_election: true
I20260929 17:47:48.806344 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4d97c139e54d00891518e67697c190" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435" is_pre_election: true
I20260929 17:47:48.806497 5712 raft_consensus.cc:2454] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:48.806497 5990 raft_consensus.cc:2454] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:48.806813 5785 leader_election.cc:302] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724, 7c4c3649994f4233aca644a8fb236435; no voters:
I20260929 17:47:48.806998 6068 raft_consensus.cc:2781] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:48.807060 6068 raft_consensus.cc:482] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:48.807086 6068 raft_consensus.cc:3037] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:48.808089 6068 raft_consensus.cc:504] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.808403 6068 leader_election.cc:288] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Requested vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131)
I20260929 17:47:48.808727 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4d97c139e54d00891518e67697c190" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435"
I20260929 17:47:48.808727 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4d97c139e54d00891518e67697c190" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724"
I20260929 17:47:48.808842 5712 raft_consensus.cc:3037] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:48.808846 5990 raft_consensus.cc:3037] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:48.809773 5990 raft_consensus.cc:2454] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:48.809991 5712 raft_consensus.cc:2454] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:48.810053 5786 leader_election.cc:302] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724; no voters:
I20260929 17:47:48.810221 6068 raft_consensus.cc:2781] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:48.810318 6068 raft_consensus.cc:686] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 LEADER]: Becoming Leader. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Running, Role: LEADER
I20260929 17:47:48.810451 6068 consensus_queue.cc:237] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.811056 6071 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: TxnStatusTablet state changed. Reason: New leader 058c1156cfd6467bbe2f320c0a9e9aea. Latest consensus state: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.811151 6071 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:47:48.811381 6083 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:47:48.811638 6083 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:47:48.812093 5554 catalog_manager.cc:5495] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea reported cstate change: term changed from 0 to 1, leader changed from <none> to 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194). New cstate: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } health_report { overall_health: UNKNOWN } } }
I20260929 17:47:48.834227 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.830153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.3.183.194:40543)
0929 17:47:48.830440 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.830445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.830459 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:47:48.830555 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.830559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.830640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.830742 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:48.830994 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.831559 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.832770 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.833031 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.833120 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.833429 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:48.833853 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:48.833940 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.834032 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.834082 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260929 17:47:48.834247 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.830060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40543 (local address 127.0.0.1:34708)
0929 17:47:48.830204 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.830220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.830326 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.830758 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.830762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.830771 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:47:48.830885 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.830892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.831648 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.831651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.832633 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.832643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.833135 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.833138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.833234 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.833288 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:47:48.833941 (+ 653us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:47:48.833952 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.834022 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.834107 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 17:47:48.839301 5712 raft_consensus.cc:1261] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:48.839329 5990 raft_consensus.cc:1261] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:48.840112 6071 consensus_queue.cc:1035] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:48.840324 6068 consensus_queue.cc:1035] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:48.843858 6067 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: TxnStatusTablet state changed. Reason: New leader 058c1156cfd6467bbe2f320c0a9e9aea. Latest consensus state: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.843989 6067 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:48.844332 6069 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: TxnStatusTablet state changed. Reason: New leader 058c1156cfd6467bbe2f320c0a9e9aea. Latest consensus state: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.844458 6069 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:48.848379 6067 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.848501 6067 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 252100efde454c8b87a20e9177c23724: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:48.849059 6088 mvcc.cc:205] Tried to move back new op lower bound from 7334723865958707200 to 7334723865848807424. Current Snapshot: MvccSnapshot[applied={T|T < 7334723865958707200}]
I20260929 17:47:48.849412 6069 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.849529 6069 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 7c4c3649994f4233aca644a8fb236435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:47:48.849851 6089 mvcc.cc:205] Tried to move back new op lower bound from 7334723865958707200 to 7334723865848807424. Current Snapshot: MvccSnapshot[applied={T|T < 7334723865958707200}]
I20260929 17:47:48.850049 6068 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.850163 6068 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:47:48.850452 6071 tablet_replica.cc:439] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } } }
I20260929 17:47:48.850551 6087 mvcc.cc:205] Tried to move back new op lower bound from 7334723865958707200 to 7334723865848807424. Current Snapshot: MvccSnapshot[applied={T|T < 7334723865958707200}]
I20260929 17:47:48.850557 6071 tablet_replica.cc:442] T cf4d97c139e54d00891518e67697c190 P 058c1156cfd6467bbe2f320c0a9e9aea: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:47:48.859663 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.856153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.3.183.194:40543)
0929 17:47:48.856272 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.856277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.856424 (+ 147us) server_negotiation.cc:407] Connection header received
0929 17:47:48.856525 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.856528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.856576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.856647 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:48.856849 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.857663 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.858542 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.858881 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.858976 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.859249 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:48.859328 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:48.859394 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.859475 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.859526 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:47:48.870991 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.867262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:49039 (local address 127.3.183.194:40543)
0929 17:47:48.867486 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:48.867491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:48.867746 (+ 255us) server_negotiation.cc:407] Connection header received
0929 17:47:48.867970 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:48.867973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:48.868014 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:48.868075 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:48.868327 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.868830 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.870005 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.870207 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.870294 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.870577 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:48.870637 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:48.870692 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:48.870771 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 17:47:48.870818 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":145}
I20260929 17:47:48.871012 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:48.867148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40543 (local address 127.3.183.194:49039)
0929 17:47:48.867680 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:48.867693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:48.867836 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:48.868095 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:48.868098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:48.868106 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:47:48.868238 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.868244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.868920 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.868923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.869885 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:48.869895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:48.870310 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:48.870314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:48.870401 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:48.870438 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:47:48.870703 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:47:48.870708 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:47:48.870764 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:47:48.870833 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":87,"threads_started":1}
I20260929 17:47:48.877862 5712 raft_consensus.cc:1261] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:48.877954 5990 raft_consensus.cc:1261] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:48.878300 6096 consensus_queue.cc:1035] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:48.878486 6092 consensus_queue.cc:1035] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:48.915560 5554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:48.915766 5554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:48.918064 5554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Sending DeleteTablet for 3 replicas of tablet 1cf7a35348224ae2a2cac8b9da28440e
I20260929 17:47:48.918730 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf7a35348224ae2a2cac8b9da28440e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:48 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45510
I20260929 17:47:48.918776 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf7a35348224ae2a2cac8b9da28440e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:48 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34244
I20260929 17:47:48.919216 6105 tablet_replica.cc:330] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: stopping tablet replica
I20260929 17:47:48.919516 6105 raft_consensus.cc:2229] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:48.919673 6106 tablet_replica.cc:330] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: stopping tablet replica
I20260929 17:47:48.919731 6105 raft_consensus.cc:2258] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:48.919768 5554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:47:48.919963 6106 raft_consensus.cc:2229] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:48.920140 6106 raft_consensus.cc:2258] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 17:47:48.920156 5554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:48.921101 6105 ts_tablet_manager.cc:1905] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:48.921340 6106 ts_tablet_manager.cc:1905] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:48.921648 5832 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf7a35348224ae2a2cac8b9da28440e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:48 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34706
I20260929 17:47:48.922111 6107 tablet_replica.cc:330] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: stopping tablet replica
I20260929 17:47:48.922384 6107 raft_consensus.cc:2229] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:47:48.922849 6107 raft_consensus.cc:2258] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:48.924093 6107 ts_tablet_manager.cc:1905] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:48.924394 6105 ts_tablet_manager.cc:1918] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:48.924489 6105 log.cc:1196] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/1cf7a35348224ae2a2cac8b9da28440e
I20260929 17:47:48.924787 6105 ts_tablet_manager.cc:1939] T 1cf7a35348224ae2a2cac8b9da28440e P 7c4c3649994f4233aca644a8fb236435: Deleting consensus metadata
I20260929 17:47:48.925917 5541 catalog_manager.cc:4841] TS 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767): tablet 1cf7a35348224ae2a2cac8b9da28440e (table test-table [id=cde91ad93adf4ebcbcadff2ff0fd57fe]) successfully deleted
I20260929 17:47:48.926309 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 45a4a3a69c924d609a714a405d947403 (DEFAULT_TABLE table=test-table [id=9219e870de4f4c3d844a9f3015afaf83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:48.926318 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 45a4a3a69c924d609a714a405d947403 (DEFAULT_TABLE table=test-table [id=9219e870de4f4c3d844a9f3015afaf83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:48.926563 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a4a3a69c924d609a714a405d947403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.926556 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a4a3a69c924d609a714a405d947403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.927350 6107 ts_tablet_manager.cc:1918] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:48.927482 6107 log.cc:1196] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/1cf7a35348224ae2a2cac8b9da28440e
I20260929 17:47:48.927902 6107 ts_tablet_manager.cc:1939] T 1cf7a35348224ae2a2cac8b9da28440e P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting consensus metadata
I20260929 17:47:48.928994 6062 tablet_bootstrap.cc:492] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: Bootstrap starting.
I20260929 17:47:48.929142 5539 catalog_manager.cc:4841] TS 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194:40543): tablet 1cf7a35348224ae2a2cac8b9da28440e (table test-table [id=cde91ad93adf4ebcbcadff2ff0fd57fe]) successfully deleted
I20260929 17:47:48.929332 5832 tablet_service.cc:1467] Processing CreateTablet for tablet 45a4a3a69c924d609a714a405d947403 (DEFAULT_TABLE table=test-table [id=9219e870de4f4c3d844a9f3015afaf83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:48.929577 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a4a3a69c924d609a714a405d947403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:48.929884 6062 tablet_bootstrap.cc:654] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.930493 6063 tablet_bootstrap.cc:492] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: Bootstrap starting.
I20260929 17:47:48.931444 6063 tablet_bootstrap.cc:654] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.931705 6061 tablet_bootstrap.cc:492] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: Bootstrap starting.
I20260929 17:47:48.932770 6061 tablet_bootstrap.cc:654] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:48.932812 6062 tablet_bootstrap.cc:492] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: No bootstrap required, opened a new log
I20260929 17:47:48.932888 6062 ts_tablet_manager.cc:1397] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260929 17:47:48.933053 6063 tablet_bootstrap.cc:492] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: No bootstrap required, opened a new log
I20260929 17:47:48.933120 6063 ts_tablet_manager.cc:1397] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:47:48.933310 6062 raft_consensus.cc:348] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.933413 6062 raft_consensus.cc:374] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.933444 6062 raft_consensus.cc:729] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c3649994f4233aca644a8fb236435, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.933519 6063 raft_consensus.cc:348] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.933560 6062 consensus_queue.cc:260] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.933607 6063 raft_consensus.cc:374] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.933655 6063 raft_consensus.cc:729] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252100efde454c8b87a20e9177c23724, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.933817 6062 ts_tablet_manager.cc:1428] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:47:48.933766 6063 consensus_queue.cc:260] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.934067 6063 ts_tablet_manager.cc:1428] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:47:48.934355 6106 ts_tablet_manager.cc:1918] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:48.934435 6106 log.cc:1196] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/1cf7a35348224ae2a2cac8b9da28440e
I20260929 17:47:48.934732 6106 ts_tablet_manager.cc:1939] T 1cf7a35348224ae2a2cac8b9da28440e P 252100efde454c8b87a20e9177c23724: Deleting consensus metadata
I20260929 17:47:48.936650 6061 tablet_bootstrap.cc:492] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: No bootstrap required, opened a new log
I20260929 17:47:48.936733 6061 ts_tablet_manager.cc:1397] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 17:47:48.937117 6061 raft_consensus.cc:348] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.937256 6061 raft_consensus.cc:374] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:48.937294 6061 raft_consensus.cc:729] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Initialized, Role: FOLLOWER
I20260929 17:47:48.937409 6061 consensus_queue.cc:260] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:48.937660 6061 ts_tablet_manager.cc:1428] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:47:48.938128 5542 catalog_manager.cc:4841] TS 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131): tablet 1cf7a35348224ae2a2cac8b9da28440e (table test-table [id=cde91ad93adf4ebcbcadff2ff0fd57fe]) successfully deleted
I20260929 17:47:49.015388 6067 raft_consensus.cc:482] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:49.015542 6067 raft_consensus.cc:504] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:49.016757 6067 leader_election.cc:288] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194:40543)
I20260929 17:47:49.021498 6113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.017197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:56481 (local address 127.3.183.193:34767)
0929 17:47:49.017641 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:49.017646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:49.017669 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:47:49.017713 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:49.017715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:49.017757 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:49.017842 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:49.018123 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.018846 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.020400 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.020927 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.021040 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.021099 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:49.021287 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 17:47:49.021364 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":54,"threads_started":1}
I20260929 17:47:49.021518 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.016996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:34767 (local address 127.3.183.195:56481)
0929 17:47:49.017398 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:49.017409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:49.017476 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:49.017869 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:49.017872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:49.017891 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:49.018005 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.018011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.018941 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.018944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.020257 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.020267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.021040 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.021043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.021176 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.021200 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:47:49.021272 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:47:49.021353 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":40,"thread_start_us":114,"threads_started":1}
I20260929 17:47:49.022320 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a4a3a69c924d609a714a405d947403" candidate_uuid: "252100efde454c8b87a20e9177c23724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435" is_pre_election: true
I20260929 17:47:49.022454 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.017333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:36731 (local address 127.3.183.194:40543)
0929 17:47:49.017446 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:49.017451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:49.017468 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:47:49.017521 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:49.017524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:49.017583 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:49.017663 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:47:49.017942 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.018699 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.021277 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.021903 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.021990 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.022036 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:49.022254 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 17:47:49.022336 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:47:49.022482 5712 raft_consensus.cc:2454] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252100efde454c8b87a20e9177c23724 in term 0.
I20260929 17:47:49.022760 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.016994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40543 (local address 127.3.183.195:36731)
0929 17:47:49.017319 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:49.017336 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:49.017439 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:49.017687 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:49.017690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:49.017708 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:47:49.017818 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.017824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.018836 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.018839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.021127 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.021136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.022031 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.022034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.022117 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.022137 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:47:49.022217 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:47:49.022285 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1091,"thread_start_us":70,"threads_started":1}
I20260929 17:47:49.022949 5925 leader_election.cc:302] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252100efde454c8b87a20e9177c23724, 7c4c3649994f4233aca644a8fb236435; no voters:
I20260929 17:47:49.023212 6067 raft_consensus.cc:2781] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:49.023272 6067 raft_consensus.cc:482] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:49.023310 6067 raft_consensus.cc:3037] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:49.023260 5852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a4a3a69c924d609a714a405d947403" candidate_uuid: "252100efde454c8b87a20e9177c23724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" is_pre_election: true
I20260929 17:47:49.023473 5852 raft_consensus.cc:2454] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252100efde454c8b87a20e9177c23724 in term 0.
I20260929 17:47:49.024410 6067 raft_consensus.cc:504] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:49.024726 6067 leader_election.cc:288] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [CANDIDATE]: Term 1 election: Requested vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194:40543)
I20260929 17:47:49.025259 5852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a4a3a69c924d609a714a405d947403" candidate_uuid: "252100efde454c8b87a20e9177c23724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058c1156cfd6467bbe2f320c0a9e9aea"
I20260929 17:47:49.025369 5852 raft_consensus.cc:3037] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:49.025362 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a4a3a69c924d609a714a405d947403" candidate_uuid: "252100efde454c8b87a20e9177c23724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435"
I20260929 17:47:49.025475 5712 raft_consensus.cc:3037] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:49.026309 5712 raft_consensus.cc:2454] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252100efde454c8b87a20e9177c23724 in term 1.
I20260929 17:47:49.026467 5852 raft_consensus.cc:2454] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252100efde454c8b87a20e9177c23724 in term 1.
I20260929 17:47:49.026824 5923 leader_election.cc:302] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724, 7c4c3649994f4233aca644a8fb236435; no voters:
I20260929 17:47:49.027005 6067 raft_consensus.cc:2781] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:49.027176 6067 raft_consensus.cc:686] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 1 LEADER]: Becoming Leader. State: Replica: 252100efde454c8b87a20e9177c23724, State: Running, Role: LEADER
I20260929 17:47:49.027418 6067 consensus_queue.cc:237] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:49.030045 5554 catalog_manager.cc:5495] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 252100efde454c8b87a20e9177c23724 (127.3.183.195). New cstate: current_term: 1 leader_uuid: "252100efde454c8b87a20e9177c23724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } health_report { overall_health: HEALTHY } } }
I20260929 17:47:49.063512 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.059620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.3.183.195:40131)
0929 17:47:49.059951 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:49.059959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:49.059970 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:47:49.060043 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:49.060045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:49.060094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:49.060239 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:49.060463 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.061049 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.062008 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.062284 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.062381 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.062672 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:49.063114 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:49.063215 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:49.063297 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 17:47:49.063351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":161,"threads_started":1}
I20260929 17:47:49.074129 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.070619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:56863 (local address 127.3.183.194:40543)
0929 17:47:49.070756 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:49.070760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:49.071012 (+ 252us) server_negotiation.cc:407] Connection header received
0929 17:47:49.071117 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:49.071119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:49.071168 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:49.071225 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:49.071470 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.072006 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.073184 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.073384 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.073449 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.073756 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:49.073820 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:49.073871 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:49.073975 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 17:47:49.074023 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:47:49.074203 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.070539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40543 (local address 127.3.183.195:56863)
0929 17:47:49.070924 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:49.070938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:49.071041 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:49.071250 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:49.071252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:49.071260 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:47:49.071378 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.071384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.072105 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.072109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.073076 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.073088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.073476 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.073480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.073578 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.073630 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:47:49.073895 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:47:49.073901 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:47:49.073965 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:47:49.074042 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":91,"threads_started":1}
I20260929 17:47:49.080271 5712 raft_consensus.cc:1261] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Refusing update from remote peer 252100efde454c8b87a20e9177c23724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:49.080503 5852 raft_consensus.cc:1261] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Refusing update from remote peer 252100efde454c8b87a20e9177c23724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:49.080734 6067 consensus_queue.cc:1035] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:49.081003 6114 consensus_queue.cc:1035] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:49.100934 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.097538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:40131 (local address 127.3.183.194:47189)
0929 17:47:49.097686 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:49.097702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:49.097782 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:49.098026 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:49.098028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:49.098036 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:47:49.098160 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.098168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.098886 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.098890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.099762 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:49.099774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.100360 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.100363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:49.100430 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.100457 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:47:49.100724 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:47:49.100727 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:47:49.100764 (+ 37us) client_negotiation.cc:240] Negotiation successful
0929 17:47:49.100829 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 17:47:49.100960 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:49.097740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:47189 (local address 127.3.183.195:40131)
0929 17:47:49.097846 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:49.097850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:49.097860 (+ 10us) server_negotiation.cc:407] Connection header received
0929 17:47:49.097899 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:49.097902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:49.097939 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:49.097996 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:49.098247 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.098772 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.099924 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:49.100249 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:49.100369 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:49.100557 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:49.100649 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:49.100726 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:49.100778 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 17:47:49.100835 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 17:47:49.101801 6088 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 17:47:50.659845 5555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:50.660017 5555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:50.662415 5555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Sending DeleteTablet for 3 replicas of tablet 45a4a3a69c924d609a714a405d947403
I20260929 17:47:50.662979 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 45a4a3a69c924d609a714a405d947403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45510
I20260929 17:47:50.662979 5832 tablet_service.cc:1514] Processing DeleteTablet for tablet 45a4a3a69c924d609a714a405d947403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34706
I20260929 17:47:50.663034 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet 45a4a3a69c924d609a714a405d947403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34244
I20260929 17:47:50.663313 6147 tablet_replica.cc:330] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: stopping tablet replica
I20260929 17:47:50.663434 6147 raft_consensus.cc:2229] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:50.663544 6147 raft_consensus.cc:2258] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:50.663707 6148 tablet_replica.cc:330] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: stopping tablet replica
I20260929 17:47:50.663877 6149 tablet_replica.cc:330] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: stopping tablet replica
I20260929 17:47:50.663888 6148 raft_consensus.cc:2229] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:50.664018 6149 raft_consensus.cc:2229] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:47:50.664124 6148 raft_consensus.cc:2258] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:50.664232 6147 ts_tablet_manager.cc:1905] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:50.664309 6149 raft_consensus.cc:2258] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:50.664785 6148 ts_tablet_manager.cc:1905] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:50.664876 6149 ts_tablet_manager.cc:1905] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:50.666901 6147 ts_tablet_manager.cc:1918] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:47:50.666982 6147 log.cc:1196] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/45a4a3a69c924d609a714a405d947403
I20260929 17:47:50.667248 6147 ts_tablet_manager.cc:1939] T 45a4a3a69c924d609a714a405d947403 P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting consensus metadata
I20260929 17:47:50.667505 6148 ts_tablet_manager.cc:1918] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:47:50.667518 6149 ts_tablet_manager.cc:1918] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:47:50.667578 6149 log.cc:1196] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/45a4a3a69c924d609a714a405d947403
I20260929 17:47:50.667691 6148 log.cc:1196] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/45a4a3a69c924d609a714a405d947403
I20260929 17:47:50.667848 6149 ts_tablet_manager.cc:1939] T 45a4a3a69c924d609a714a405d947403 P 252100efde454c8b87a20e9177c23724: Deleting consensus metadata
I20260929 17:47:50.668049 6148 ts_tablet_manager.cc:1939] T 45a4a3a69c924d609a714a405d947403 P 7c4c3649994f4233aca644a8fb236435: Deleting consensus metadata
I20260929 17:47:50.668012 5539 catalog_manager.cc:4841] TS 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194:40543): tablet 45a4a3a69c924d609a714a405d947403 (table test-table [id=9219e870de4f4c3d844a9f3015afaf83]) successfully deleted
I20260929 17:47:50.668486 5542 catalog_manager.cc:4841] TS 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131): tablet 45a4a3a69c924d609a714a405d947403 (table test-table [id=9219e870de4f4c3d844a9f3015afaf83]) successfully deleted
I20260929 17:47:50.668708 5541 catalog_manager.cc:4841] TS 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767): tablet 45a4a3a69c924d609a714a405d947403 (table test-table [id=9219e870de4f4c3d844a9f3015afaf83]) successfully deleted
Sep 29 17:47:50 dist-test-slave-wbpt krb5kdc[5510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:47:50 dist-test-slave-wbpt krb5kdc[5510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:50.691126 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:50.682915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.3.183.254:34143)
0929 17:47:50.683585 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:50.683611 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:50.683625 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:47:50.683663 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:50.683666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:50.683711 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:50.683809 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:50.684460 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:50.685068 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:50.685852 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:50.686086 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:50.686160 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:50.688293 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:50.688296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:50.688298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:50.688322 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:50.689803 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:50.690149 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:50.690151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:50.690153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:50.690223 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:50.690424 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:50.690426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:50.690427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:50.690622 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:50.690715 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:50.690897 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 17:47:50.690996 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":97,"threads_started":1}
I20260929 17:47:50.693678 5555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:47:50.694070 5555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:50.698390 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 44b574c2002e40348d4d75bb6698eb07 (DEFAULT_TABLE table=test-table [id=39ad6c8cfd794a55acdfd9b6869163e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:50.698519 5832 tablet_service.cc:1467] Processing CreateTablet for tablet 44b574c2002e40348d4d75bb6698eb07 (DEFAULT_TABLE table=test-table [id=39ad6c8cfd794a55acdfd9b6869163e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:50.698575 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 44b574c2002e40348d4d75bb6698eb07 (DEFAULT_TABLE table=test-table [id=39ad6c8cfd794a55acdfd9b6869163e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:50.698711 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b574c2002e40348d4d75bb6698eb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:50.698791 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b574c2002e40348d4d75bb6698eb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:50.698820 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b574c2002e40348d4d75bb6698eb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:50.700861 6158 tablet_bootstrap.cc:492] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: Bootstrap starting.
I20260929 17:47:50.701186 6159 tablet_bootstrap.cc:492] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: Bootstrap starting.
I20260929 17:47:50.701678 6160 tablet_bootstrap.cc:492] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: Bootstrap starting.
I20260929 17:47:50.702066 6159 tablet_bootstrap.cc:654] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:50.702139 6158 tablet_bootstrap.cc:654] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:50.702584 6160 tablet_bootstrap.cc:654] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:50.703263 6159 tablet_bootstrap.cc:492] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: No bootstrap required, opened a new log
I20260929 17:47:50.703352 6159 ts_tablet_manager.cc:1397] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:50.703418 6158 tablet_bootstrap.cc:492] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: No bootstrap required, opened a new log
I20260929 17:47:50.703509 6158 ts_tablet_manager.cc:1397] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:47:50.703712 6160 tablet_bootstrap.cc:492] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: No bootstrap required, opened a new log
I20260929 17:47:50.703819 6160 ts_tablet_manager.cc:1397] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:50.703928 6158 raft_consensus.cc:348] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.703928 6159 raft_consensus.cc:348] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.704030 6158 raft_consensus.cc:374] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:50.704036 6159 raft_consensus.cc:374] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:50.704063 6158 raft_consensus.cc:729] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Initialized, Role: FOLLOWER
I20260929 17:47:50.704066 6159 raft_consensus.cc:729] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252100efde454c8b87a20e9177c23724, State: Initialized, Role: FOLLOWER
I20260929 17:47:50.704176 6158 consensus_queue.cc:260] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.704176 6159 consensus_queue.cc:260] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.704244 6160 raft_consensus.cc:348] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.704347 6160 raft_consensus.cc:374] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:50.704378 6160 raft_consensus.cc:729] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c3649994f4233aca644a8fb236435, State: Initialized, Role: FOLLOWER
I20260929 17:47:50.704428 6159 ts_tablet_manager.cc:1428] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:47:50.704504 6158 ts_tablet_manager.cc:1428] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:47:50.704509 6160 consensus_queue.cc:260] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.704854 6160 ts_tablet_manager.cc:1428] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:47:50.773802 6133 raft_consensus.cc:482] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:50.773949 6133 raft_consensus.cc:504] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.774396 6133 leader_election.cc:288] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131)
I20260929 17:47:50.774953 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b574c2002e40348d4d75bb6698eb07" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724" is_pre_election: true
I20260929 17:47:50.775164 5990 raft_consensus.cc:2454] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:50.775612 5786 leader_election.cc:302] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724; no voters:
I20260929 17:47:50.775563 5711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b574c2002e40348d4d75bb6698eb07" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435" is_pre_election: true
I20260929 17:47:50.775753 5711 raft_consensus.cc:2454] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:50.775851 6133 raft_consensus.cc:2781] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:50.775926 6133 raft_consensus.cc:482] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:50.775957 6133 raft_consensus.cc:3037] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:50.777213 6133 raft_consensus.cc:504] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.777565 6133 leader_election.cc:288] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Requested vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131)
I20260929 17:47:50.777905 5711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b574c2002e40348d4d75bb6698eb07" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435"
I20260929 17:47:50.777928 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b574c2002e40348d4d75bb6698eb07" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724"
I20260929 17:47:50.778029 5711 raft_consensus.cc:3037] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:50.778035 5990 raft_consensus.cc:3037] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:50.779086 5990 raft_consensus.cc:2454] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:50.779209 5711 raft_consensus.cc:2454] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:50.779361 5786 leader_election.cc:302] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724; no voters:
I20260929 17:47:50.779532 6133 raft_consensus.cc:2781] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:50.779659 6133 raft_consensus.cc:686] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 LEADER]: Becoming Leader. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Running, Role: LEADER
I20260929 17:47:50.779825 6133 consensus_queue.cc:237] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.781455 5554 catalog_manager.cc:5495] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea reported cstate change: term changed from 0 to 1, leader changed from <none> to 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194). New cstate: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } health_report { overall_health: UNKNOWN } } }
I20260929 17:47:50.810444 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:50.807537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.3.183.194:40543)
0929 17:47:50.807825 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:50.807835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:50.807850 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:47:50.807895 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:50.807897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:50.807949 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:50.808022 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:47:50.808214 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:50.808754 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:50.809559 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:50.809798 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:50.809882 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:50.810064 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:47:50.810131 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:47:50.810181 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:50.810239 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 17:47:50.810290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
I20260929 17:47:50.815892 5990 raft_consensus.cc:1261] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:50.815936 5711 raft_consensus.cc:1261] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:50.816291 6128 consensus_queue.cc:1035] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:50.816540 6128 consensus_queue.cc:1035] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:50.846441 5552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:50.846581 5552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:50.848384 5552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Sending DeleteTablet for 3 replicas of tablet 44b574c2002e40348d4d75bb6698eb07
I20260929 17:47:50.848886 5832 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b574c2002e40348d4d75bb6698eb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34706
I20260929 17:47:50.848948 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b574c2002e40348d4d75bb6698eb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34244
I20260929 17:47:50.849036 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b574c2002e40348d4d75bb6698eb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45510
I20260929 17:47:50.849143 6147 tablet_replica.cc:330] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: stopping tablet replica
I20260929 17:47:50.849202 6148 tablet_replica.cc:330] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: stopping tablet replica
I20260929 17:47:50.849277 6147 raft_consensus.cc:2229] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:47:50.849313 6148 raft_consensus.cc:2229] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:50.849411 6148 raft_consensus.cc:2258] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:50.849474 6147 raft_consensus.cc:2258] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:50.849555 6149 tablet_replica.cc:330] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: stopping tablet replica
I20260929 17:47:50.849665 6149 raft_consensus.cc:2229] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:50.849766 6149 raft_consensus.cc:2258] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:50.850256 6147 ts_tablet_manager.cc:1905] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:50.850256 6148 ts_tablet_manager.cc:1905] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:50.850476 6149 ts_tablet_manager.cc:1905] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:50.852572 6149 ts_tablet_manager.cc:1918] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:50.852643 6149 log.cc:1196] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/44b574c2002e40348d4d75bb6698eb07
I20260929 17:47:50.852746 6148 ts_tablet_manager.cc:1918] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:50.852772 6147 ts_tablet_manager.cc:1918] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:47:50.852828 6147 log.cc:1196] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/44b574c2002e40348d4d75bb6698eb07
I20260929 17:47:50.852824 6148 log.cc:1196] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/44b574c2002e40348d4d75bb6698eb07
I20260929 17:47:50.852885 6149 ts_tablet_manager.cc:1939] T 44b574c2002e40348d4d75bb6698eb07 P 252100efde454c8b87a20e9177c23724: Deleting consensus metadata
I20260929 17:47:50.853078 6147 ts_tablet_manager.cc:1939] T 44b574c2002e40348d4d75bb6698eb07 P 058c1156cfd6467bbe2f320c0a9e9aea: Deleting consensus metadata
I20260929 17:47:50.853078 6148 ts_tablet_manager.cc:1939] T 44b574c2002e40348d4d75bb6698eb07 P 7c4c3649994f4233aca644a8fb236435: Deleting consensus metadata
I20260929 17:47:50.849781 5552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:47:50.853704 5539 catalog_manager.cc:4841] TS 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194:40543): tablet 44b574c2002e40348d4d75bb6698eb07 (table test-table [id=39ad6c8cfd794a55acdfd9b6869163e0]) successfully deleted
W20260929 17:47:50.853739 5552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:47:50.854151 5542 catalog_manager.cc:4841] TS 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131): tablet 44b574c2002e40348d4d75bb6698eb07 (table test-table [id=39ad6c8cfd794a55acdfd9b6869163e0]) successfully deleted
I20260929 17:47:50.854384 5541 catalog_manager.cc:4841] TS 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767): tablet 44b574c2002e40348d4d75bb6698eb07 (table test-table [id=39ad6c8cfd794a55acdfd9b6869163e0]) successfully deleted
I20260929 17:47:50.858022 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 98f7771a13aa43e1869b9c0d19858840 (DEFAULT_TABLE table=test-table [id=65cf9b01dc464d98acd6b5bf6b624cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:50.858112 5832 tablet_service.cc:1467] Processing CreateTablet for tablet 98f7771a13aa43e1869b9c0d19858840 (DEFAULT_TABLE table=test-table [id=65cf9b01dc464d98acd6b5bf6b624cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:50.858157 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 98f7771a13aa43e1869b9c0d19858840 (DEFAULT_TABLE table=test-table [id=65cf9b01dc464d98acd6b5bf6b624cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:47:50.858299 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f7771a13aa43e1869b9c0d19858840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:50.858312 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f7771a13aa43e1869b9c0d19858840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:50.858389 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f7771a13aa43e1869b9c0d19858840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:50.860198 6159 tablet_bootstrap.cc:492] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: Bootstrap starting.
I20260929 17:47:50.860635 6158 tablet_bootstrap.cc:492] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea: Bootstrap starting.
I20260929 17:47:50.860822 6160 tablet_bootstrap.cc:492] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435: Bootstrap starting.
I20260929 17:47:50.861114 6159 tablet_bootstrap.cc:654] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:50.861429 6158 tablet_bootstrap.cc:654] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:50.862182 6160 tablet_bootstrap.cc:654] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:50.862198 6159 tablet_bootstrap.cc:492] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: No bootstrap required, opened a new log
I20260929 17:47:50.862277 6159 ts_tablet_manager.cc:1397] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:47:50.862651 6159 raft_consensus.cc:348] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.862746 6159 raft_consensus.cc:374] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:50.862766 6159 raft_consensus.cc:729] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252100efde454c8b87a20e9177c23724, State: Initialized, Role: FOLLOWER
I20260929 17:47:50.862861 6159 consensus_queue.cc:260] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.863106 6159 ts_tablet_manager.cc:1428] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:47:50.863879 6158 tablet_bootstrap.cc:492] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea: No bootstrap required, opened a new log
I20260929 17:47:50.863963 6158 ts_tablet_manager.cc:1397] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:47:50.864073 6160 tablet_bootstrap.cc:492] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435: No bootstrap required, opened a new log
I20260929 17:47:50.864185 6160 ts_tablet_manager.cc:1397] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260929 17:47:50.864365 6158 raft_consensus.cc:348] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.864462 6158 raft_consensus.cc:374] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:50.864496 6158 raft_consensus.cc:729] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Initialized, Role: FOLLOWER
I20260929 17:47:50.864611 6158 consensus_queue.cc:260] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.864848 6158 ts_tablet_manager.cc:1428] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:47:50.864853 6160 raft_consensus.cc:348] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.865018 6160 raft_consensus.cc:374] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:50.865063 6160 raft_consensus.cc:729] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c3649994f4233aca644a8fb236435, State: Initialized, Role: FOLLOWER
I20260929 17:47:50.865207 6160 consensus_queue.cc:260] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.866099 6160 ts_tablet_manager.cc:1428] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:50.969173 6125 raft_consensus.cc:482] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:47:50.969314 6125 raft_consensus.cc:504] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.969758 6125 leader_election.cc:288] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131)
I20260929 17:47:50.970229 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f7771a13aa43e1869b9c0d19858840" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724" is_pre_election: true
I20260929 17:47:50.970229 5711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f7771a13aa43e1869b9c0d19858840" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435" is_pre_election: true
I20260929 17:47:50.970387 5990 raft_consensus.cc:2454] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:50.970387 5711 raft_consensus.cc:2454] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 0.
I20260929 17:47:50.970724 5785 leader_election.cc:302] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724, 7c4c3649994f4233aca644a8fb236435; no voters:
I20260929 17:47:50.970932 6125 raft_consensus.cc:2781] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:47:50.970999 6125 raft_consensus.cc:482] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:47:50.971028 6125 raft_consensus.cc:3037] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:50.972045 6125 raft_consensus.cc:504] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.972388 6125 leader_election.cc:288] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Requested vote from peers 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767), 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131)
I20260929 17:47:50.972749 5711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f7771a13aa43e1869b9c0d19858840" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4c3649994f4233aca644a8fb236435"
I20260929 17:47:50.972779 5990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f7771a13aa43e1869b9c0d19858840" candidate_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252100efde454c8b87a20e9177c23724"
I20260929 17:47:50.972867 5711 raft_consensus.cc:3037] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:50.972890 5990 raft_consensus.cc:3037] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:50.973881 5990 raft_consensus.cc:2454] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:50.974079 5711 raft_consensus.cc:2454] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c1156cfd6467bbe2f320c0a9e9aea in term 1.
I20260929 17:47:50.974157 5786 leader_election.cc:302] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c1156cfd6467bbe2f320c0a9e9aea, 252100efde454c8b87a20e9177c23724; no voters:
I20260929 17:47:50.974319 6125 raft_consensus.cc:2781] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:50.974424 6125 raft_consensus.cc:686] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 LEADER]: Becoming Leader. State: Replica: 058c1156cfd6467bbe2f320c0a9e9aea, State: Running, Role: LEADER
I20260929 17:47:50.974565 6125 consensus_queue.cc:237] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } }
I20260929 17:47:50.976220 5552 catalog_manager.cc:5495] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea reported cstate change: term changed from 0 to 1, leader changed from <none> to 058c1156cfd6467bbe2f320c0a9e9aea (127.3.183.194). New cstate: current_term: 1 leader_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058c1156cfd6467bbe2f320c0a9e9aea" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 } health_report { overall_health: UNKNOWN } } }
I20260929 17:47:50.997602 5990 raft_consensus.cc:1261] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:50.997676 5711 raft_consensus.cc:1261] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Refusing update from remote peer 058c1156cfd6467bbe2f320c0a9e9aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:47:50.997983 6133 consensus_queue.cc:1035] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "252100efde454c8b87a20e9177c23724" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:47:50.998138 6125 consensus_queue.cc:1035] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4c3649994f4233aca644a8fb236435" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 17:47:51.010715 6126 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 17:47:52.559093 5553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:52.559260 5553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:47:52.561553 5553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb84761044a343c39e579c3965709f6d: Sending DeleteTablet for 3 replicas of tablet 98f7771a13aa43e1869b9c0d19858840
I20260929 17:47:52.562026 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f7771a13aa43e1869b9c0d19858840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:52 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45510
I20260929 17:47:52.562239 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f7771a13aa43e1869b9c0d19858840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:52 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34244
I20260929 17:47:52.562333 6197 tablet_replica.cc:330] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435: stopping tablet replica
I20260929 17:47:52.562314 5832 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f7771a13aa43e1869b9c0d19858840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:47:52 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34706
I20260929 17:47:52.562445 6197 raft_consensus.cc:2229] T 98f7771a13aa43e1869b9c0d19858840 P 7c4c3649994f4233aca644a8fb236435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:52.562448 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 5627
W20260929 17:47:52.567525 5541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:34767 (error 108)
I20260929 17:47:52.567567 6199 tablet_replica.cc:330] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea: stopping tablet replica
I20260929 17:47:52.567607 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 5767
I20260929 17:47:52.567701 6199 raft_consensus.cc:2229] T 98f7771a13aa43e1869b9c0d19858840 P 058c1156cfd6467bbe2f320c0a9e9aea [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:47:52.567656 6198 tablet_replica.cc:330] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: stopping tablet replica
I20260929 17:47:52.568341 6198 raft_consensus.cc:2229] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:47:52.568451 6198 raft_consensus.cc:2258] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:47:52.569077 6198 ts_tablet_manager.cc:1905] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:47:52.569713 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:52.569163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:34767 (local address 127.0.0.1:45526)
0929 17:47:52.569530 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:52.569608 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":72,"threads_started":1}
W20260929 17:47:52.569849 5541 catalog_manager.cc:4584] TS 7c4c3649994f4233aca644a8fb236435 (127.3.183.193:34767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98f7771a13aa43e1869b9c0d19858840: Network error: Client connection negotiation failed: client connection to 127.3.183.193:34767: connect: Connection refused (error 111)
I20260929 17:47:52.571382 6198 ts_tablet_manager.cc:1918] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:47:52.571449 6198 log.cc:1196] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/98f7771a13aa43e1869b9c0d19858840
I20260929 17:47:52.571645 6198 ts_tablet_manager.cc:1939] T 98f7771a13aa43e1869b9c0d19858840 P 252100efde454c8b87a20e9177c23724: Deleting consensus metadata
I20260929 17:47:52.572372 5542 catalog_manager.cc:4841] TS 252100efde454c8b87a20e9177c23724 (127.3.183.195:40131): tablet 98f7771a13aa43e1869b9c0d19858840 (table test-table [id=65cf9b01dc464d98acd6b5bf6b624cba]) successfully deleted
I20260929 17:47:52.573863 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 5907
I20260929 17:47:52.574663 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:52.573890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40543 (local address 127.0.0.1:34754)
0929 17:47:52.574523 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:52.574577 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556}
I20260929 17:47:52.579115 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 5523
2026-09-29T17:47:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6307 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:52 dist-test-slave-wbpt krb5kdc[6204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:52 dist-test-slave-wbpt krb5kdc[6204](info): set up 2 sockets
Sep 29 17:47:52 dist-test-slave-wbpt krb5kdc[6204](info): commencing operation
krb5kdc: starting...
W20260929 17:47:53.539511 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:53 dist-test-slave-wbpt krb5kdc[6204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:53.656222 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35873
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:35873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:53.764696 6217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:53.764972 6217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:53.765013 6217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:53.769021 6217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:53.769080 6217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:53.769099 6217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:53.769114 6217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:53.769129 6217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:53.774451 6217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:35873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:53.775569 6217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:53.776655 6217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:47:53.785684 6217 server_base.cc:1034] running on GCE node
W20260929 17:47:53.785612 6222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:53.785609 6225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:53.785612 6223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:53.786424 6217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:53.787566 6217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:53.788753 6217 hybrid_clock.cc:648] HybridClock initialized: now 1790704073788721 us; error 40 us; skew 500 ppm
Sep 29 17:47:53 dist-test-slave-wbpt krb5kdc[6204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:53.797005 6217 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:53.798061 6217 webserver.cc:460] Webserver started at http://127.3.183.254:34099/ using document root <none> and password file <none>
I20260929 17:47:53.798623 6217 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:53.798691 6217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:53.798880 6217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:53.801024 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6"
format_stamp: "Formatted at 2026-09-29 17:47:53 on dist-test-slave-wbpt"
server_key: "a682b77243d63a9b88a2ee4301aa8f5a"
server_key_iv: "bd475f257e39e13e37e81a1fa3079496"
server_key_version: "encryptionkey@0"
I20260929 17:47:53.801573 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6"
format_stamp: "Formatted at 2026-09-29 17:47:53 on dist-test-slave-wbpt"
server_key: "a682b77243d63a9b88a2ee4301aa8f5a"
server_key_iv: "bd475f257e39e13e37e81a1fa3079496"
server_key_version: "encryptionkey@0"
I20260929 17:47:53.805282 6217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:47:53.807600 6232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:53.808928 6217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:47:53.809057 6217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6"
format_stamp: "Formatted at 2026-09-29 17:47:53 on dist-test-slave-wbpt"
server_key: "a682b77243d63a9b88a2ee4301aa8f5a"
server_key_iv: "bd475f257e39e13e37e81a1fa3079496"
server_key_version: "encryptionkey@0"
I20260929 17:47:53.809145 6217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:53.843057 6217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:53.845906 6217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:53.846084 6217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:53.852756 6217 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:35873
I20260929 17:47:53.852782 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:35873 every 8 connection(s)
I20260929 17:47:53.853717 6217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:53.856364 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:53.861730 6285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6: Bootstrap starting.
I20260929 17:47:53.862632 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 6217
I20260929 17:47:53.862735 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:53.863008 3807 external_mini_cluster.cc:1428] Setting key 8ca89d5869fc10b1a288c4692b80a570
I20260929 17:47:53.864581 6285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:53.865255 6285 log.cc:824] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:53.867324 6285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6: No bootstrap required, opened a new log
I20260929 17:47:53.871423 6285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35873 } }
I20260929 17:47:53.871766 6285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:53.871862 6285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c26d5f4fae74473b2d15dfb0aad9fc6, State: Initialized, Role: FOLLOWER
Sep 29 17:47:53 dist-test-slave-wbpt krb5kdc[6204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:53.872346 6285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35873 } }
I20260929 17:47:53.872529 6285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:53.872603 6285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:53.872707 6285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:53.873891 6285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35873 } }
I20260929 17:47:53.874289 6285 leader_election.cc:302] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c26d5f4fae74473b2d15dfb0aad9fc6; no voters:
I20260929 17:47:53.874696 6285 leader_election.cc:288] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:53.874780 6290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:53.875017 6290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 5c26d5f4fae74473b2d15dfb0aad9fc6, State: Running, Role: LEADER
I20260929 17:47:53.875356 6290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35873 } }
I20260929 17:47:53.876091 6285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:53.877285 6292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c26d5f4fae74473b2d15dfb0aad9fc6. Latest consensus state: current_term: 1 leader_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35873 } } }
I20260929 17:47:53.877390 6292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:53.877655 6291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c26d5f4fae74473b2d15dfb0aad9fc6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35873 } } }
I20260929 17:47:53.877740 6291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:53.877857 6299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:53.878600 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:53.864180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.3.183.254:35873)
0929 17:47:53.864567 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:53.864583 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:53.864617 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:47:53.865262 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:53.865279 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:53.865681 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:53.866033 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:53.866844 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:53.867987 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:53.869141 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:53.869664 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:53.869788 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:53.872685 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:53.872693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:53.872711 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:53.872748 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:53.874648 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:53.875311 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:53.875317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:53.875323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:53.875461 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:53.875809 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:53.875813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:53.875815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:53.876377 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:53.876587 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:53.877225 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 17:47:53.877547 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260929 17:47:53.881072 6299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:53.886425 6299 catalog_manager.cc:1347] Generated new cluster ID: a158d12579594988b0ef48ffbbbc9728
I20260929 17:47:53.886488 6299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:53.897886 6299 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:53.898824 6299 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:53.911535 6299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c26d5f4fae74473b2d15dfb0aad9fc6: Generated new TSK 0
I20260929 17:47:53.912346 6299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:47:53.942484 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35873
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:54.076745 6311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:54.077045 6311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:54.077100 6311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:54.081280 6311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:54.081347 6311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:54.081430 6311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:47:54.086884 6311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:54.088066 6311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:54.089110 6311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:54.098774 6316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:54.098791 6317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:54.098934 6311 server_base.cc:1034] running on GCE node
W20260929 17:47:54.098875 6319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:54.099458 6311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:54.100189 6311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:54.101357 6311 hybrid_clock.cc:648] HybridClock initialized: now 1790704074101336 us; error 34 us; skew 500 ppm
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:54.104287 6311 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:47:54.105189 6311 webserver.cc:460] Webserver started at http://127.3.183.193:40011/ using document root <none> and password file <none>
I20260929 17:47:54.105756 6311 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:54.105825 6311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:54.106055 6311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:54.108263 6311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "8af433d52bef45d28412a3dd46e933b8"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "1ef0ffd173d77d72d6806e7479b2e09c"
server_key_iv: "24395d516cea87e341ed29e3bda2612d"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.108824 6311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "8af433d52bef45d28412a3dd46e933b8"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "1ef0ffd173d77d72d6806e7479b2e09c"
server_key_iv: "24395d516cea87e341ed29e3bda2612d"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.112831 6311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:47:54.115361 6326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.116696 6311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:54.116804 6311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "8af433d52bef45d28412a3dd46e933b8"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "1ef0ffd173d77d72d6806e7479b2e09c"
server_key_iv: "24395d516cea87e341ed29e3bda2612d"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.116910 6311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:54.135264 6311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:54.138727 6311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:54.138893 6311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:54.139468 6311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:54.140316 6311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:54.140364 6311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.140403 6311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:54.140420 6311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.149343 6311 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:40755
I20260929 17:47:54.149389 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:40755 every 8 connection(s)
I20260929 17:47:54.150259 6311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:54.159741 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 6311
I20260929 17:47:54.159868 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:47:54.160130 3807 external_mini_cluster.cc:1428] Setting key 34dad5fb59fd5758fcaa445e5398cab6
I20260929 17:47:54.164059 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:54.152097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:44933 (local address 127.3.183.254:35873)
0929 17:47:54.152242 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:54.152245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:54.153094 (+ 849us) server_negotiation.cc:407] Connection header received
0929 17:47:54.154465 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:54.154468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:54.154528 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:54.154601 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:54.155865 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.156389 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.157524 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.157761 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.157848 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:54.160387 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:54.160390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:54.160392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:54.160416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:54.161965 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:54.162486 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:54.162490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:54.162491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:54.162596 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:54.162925 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:54.162929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:54.162930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:54.163119 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:54.163205 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:54.163725 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 17:47:54.163892 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:47:54.164917 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:54.152506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35873 (local address 127.3.183.193:44933)
0929 17:47:54.152938 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:54.152982 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:54.154240 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:54.154803 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:54.154811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:54.155209 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:54.155710 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:54.155724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.156508 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.156510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:54.157411 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:54.157420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.157901 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.157905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:54.158072 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:54.158634 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:54.158662 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:54.160222 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:54.162088 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:54.162094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:54.162114 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:54.162382 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:54.162678 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:54.162682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:54.162684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:54.162839 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:54.163219 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:54.163226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:54.163488 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 17:47:54.163667 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 17:47:54.164056 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260929 17:47:54.166649 6440 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35873
I20260929 17:47:54.166890 6440 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:54.167402 6440 heartbeater.cc:511] Master 127.3.183.254:35873 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:47:54.169003 6249 ts_manager.cc:194] Registered new tserver with Master: 8af433d52bef45d28412a3dd46e933b8 (127.3.183.193:40755)
I20260929 17:47:54.170164 6249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:44933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:47:54.186339 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35873
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:54.297895 6445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:54.298156 6445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:54.298197 6445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:54.302273 6445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:54.302335 6445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:54.302395 6445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:47:54.307605 6445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:54.308624 6445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:54.309644 6445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:54.320827 6450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:54.320822 6453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:54.320883 6445 server_base.cc:1034] running on GCE node
W20260929 17:47:54.321092 6451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:54.321480 6445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:54.322129 6445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:54.323338 6445 hybrid_clock.cc:648] HybridClock initialized: now 1790704074323318 us; error 33 us; skew 500 ppm
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:54.326108 6445 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:47:54.326931 6445 webserver.cc:460] Webserver started at http://127.3.183.194:37521/ using document root <none> and password file <none>
I20260929 17:47:54.327461 6445 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:54.327525 6445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:54.327713 6445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:54.329764 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "679a43286d654233a94a77eb0d61463f"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "e58183f08a157cd0481ca8231e5f4933"
server_key_iv: "053f2dc474233d5ee94f6465e96ebda3"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.330307 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "679a43286d654233a94a77eb0d61463f"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "e58183f08a157cd0481ca8231e5f4933"
server_key_iv: "053f2dc474233d5ee94f6465e96ebda3"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.333900 6445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:47:54.336251 6460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.337476 6445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:54.337577 6445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "679a43286d654233a94a77eb0d61463f"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "e58183f08a157cd0481ca8231e5f4933"
server_key_iv: "053f2dc474233d5ee94f6465e96ebda3"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.337668 6445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:54.367231 6445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:54.370060 6445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:54.370231 6445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:54.370792 6445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:54.371577 6445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:54.371622 6445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.371675 6445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:54.371699 6445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.380421 6445 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:39607
I20260929 17:47:54.380546 6573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:39607 every 8 connection(s)
I20260929 17:47:54.381287 6445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:47:54.382505 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 6445
I20260929 17:47:54.382604 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:47:54.382879 3807 external_mini_cluster.cc:1428] Setting key cfaba9daa03f56fa6236820934756319
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:47:54.395251 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:54.383112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:44095 (local address 127.3.183.254:35873)
0929 17:47:54.383237 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:54.383242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:54.384202 (+ 960us) server_negotiation.cc:407] Connection header received
0929 17:47:54.385937 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:54.385940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:54.385992 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:54.386094 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:54.387226 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.388008 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.389151 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.389496 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.389609 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:54.391946 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:54.391950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:54.391952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:54.391976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:54.393281 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:54.393820 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:54.393823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:54.393824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:54.393899 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:54.394194 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:54.394198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:54.394199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:54.394408 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:54.394502 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:54.394965 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 17:47:54.395087 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:47:54.396129 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:54.383538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35873 (local address 127.3.183.194:44095)
0929 17:47:54.384062 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:54.384120 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:54.385690 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:54.386276 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:54.386283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:54.386629 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:54.387089 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:54.387102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.388125 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.388127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:54.389030 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:54.389040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.389645 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.389649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:54.389792 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:54.390318 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:54.390334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:54.391770 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:54.393384 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:54.393389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:54.393401 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:54.393730 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:54.393964 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:54.393966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:54.393967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:54.394112 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:54.394503 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:54.394510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:54.394739 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 17:47:54.394903 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 17:47:54.395216 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":149,"threads_started":1}
I20260929 17:47:54.397240 6574 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35873
I20260929 17:47:54.397459 6574 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:54.397925 6574 heartbeater.cc:511] Master 127.3.183.254:35873 requested a full tablet report, sending...
I20260929 17:47:54.398898 6249 ts_manager.cc:194] Registered new tserver with Master: 679a43286d654233a94a77eb0d61463f (127.3.183.194:39607)
I20260929 17:47:54.399469 6249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:44095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:47:54.410200 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35873
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:54.516687 6579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:54.516918 6579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:54.516976 6579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:54.520941 6579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:54.520995 6579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:54.521067 6579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:47:54.526219 6579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:54.527220 6579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:54.528234 6579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:54.537549 6587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:54.537521 6584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:54.537606 6579 server_base.cc:1034] running on GCE node
W20260929 17:47:54.537551 6585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:54.538183 6579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:54.538782 6579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:54.539934 6579 hybrid_clock.cc:648] HybridClock initialized: now 1790704074539913 us; error 36 us; skew 500 ppm
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:54.542594 6579 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:47:54.543394 6579 webserver.cc:460] Webserver started at http://127.3.183.195:39961/ using document root <none> and password file <none>
I20260929 17:47:54.543956 6579 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:54.544023 6579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:54.544207 6579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:54.546175 6579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "3207424d33274e7bb3542dcdceb179dc"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "ddae168566884bfe1ae12d293e96db19"
server_key_iv: "89b784af92c32a8b69d8b581a17470b1"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.546701 6579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "3207424d33274e7bb3542dcdceb179dc"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "ddae168566884bfe1ae12d293e96db19"
server_key_iv: "89b784af92c32a8b69d8b581a17470b1"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.550199 6579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:47:54.552491 6594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.553658 6579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:54.553761 6579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "3207424d33274e7bb3542dcdceb179dc"
format_stamp: "Formatted at 2026-09-29 17:47:54 on dist-test-slave-wbpt"
server_key: "ddae168566884bfe1ae12d293e96db19"
server_key_iv: "89b784af92c32a8b69d8b581a17470b1"
server_key_version: "encryptionkey@0"
I20260929 17:47:54.553859 6579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:54.571805 6579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:54.574627 6579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:54.574800 6579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:54.575368 6579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:54.576283 6579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:54.576331 6579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.576383 6579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:54.576407 6579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:54.584769 6579 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:41629
I20260929 17:47:54.584800 6707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:41629 every 8 connection(s)
I20260929 17:47:54.585625 6579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:47:54.585898 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 6579
I20260929 17:47:54.586006 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:47:54.586308 3807 external_mini_cluster.cc:1428] Setting key f7843caf4ca261d430cb070314bcf133
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:54.598624 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:54.587373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:56571 (local address 127.3.183.254:35873)
0929 17:47:54.587496 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:54.587500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:54.588576 (+ 1076us) server_negotiation.cc:407] Connection header received
0929 17:47:54.589855 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:54.589858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:54.589907 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:54.589972 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:54.591092 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.591659 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.592840 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.593082 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.593170 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:54.595546 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:54.595550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:54.595552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:54.595577 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:54.596777 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:54.597302 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:54.597304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:54.597305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:54.597381 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:54.597651 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:54.597654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:54.597655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:54.597823 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:54.597893 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:54.598361 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 17:47:54.598494 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:47:54.599591 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:54.587920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35873 (local address 127.3.183.195:56571)
0929 17:47:54.588433 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:54.588480 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:54.589669 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:54.590133 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:54.590140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:54.590484 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:54.590951 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:54.590965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.591768 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.591771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:54.592712 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:54.592722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:54.593215 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:54.593218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:54.593361 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:54.593877 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:54.593893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:54.595383 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:54.596888 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:54.596893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:54.596906 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:54.597203 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:54.597446 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:54.597448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:54.597449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:54.597576 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:54.597894 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:54.597899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:54.598147 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 17:47:54.598298 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 17:47:54.598595 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20260929 17:47:54.600648 6708 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35873
I20260929 17:47:54.600876 6708 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:54.601342 6708 heartbeater.cc:511] Master 127.3.183.254:35873 requested a full tablet report, sending...
I20260929 17:47:54.602315 6248 ts_manager.cc:194] Registered new tserver with Master: 3207424d33274e7bb3542dcdceb179dc (127.3.183.195:41629)
I20260929 17:47:54.602860 6248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:56571
I20260929 17:47:54.611877 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 17:47:54.624545 6288 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:47:54.623136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.3.183.254:35873)
0929 17:47:54.623263 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:54.623267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:54.623430 (+ 163us) server_negotiation.cc:407] Connection header received
0929 17:47:54.623543 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:54.623546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:54.623589 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:54.623663 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:54.624406 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56198: BlockingRecv error: recv got EOF from 127.0.0.1:56198 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:47:54.625697 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 6311
I20260929 17:47:54.630811 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 6445
I20260929 17:47:54.636407 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 6579
I20260929 17:47:54.640935 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 6217
2026-09-29T17:47:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2061 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6720](info): set up 2 sockets
Sep 29 17:47:54 dist-test-slave-wbpt krb5kdc[6720](info): commencing operation
krb5kdc: starting...
W20260929 17:47:55.591411 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:55 dist-test-slave-wbpt krb5kdc[6720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:55.704751 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:43029
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:43029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:55.813529 6733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:55.813798 6733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:55.813838 6733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:55.817831 6733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:55.817890 6733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:55.817909 6733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:55.817924 6733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:55.817937 6733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:55.823201 6733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:43029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:43029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:55.824270 6733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:55.825285 6733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:47:55.833592 6733 server_base.cc:1034] running on GCE node
W20260929 17:47:55.833560 6739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:55.835708 6741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:55.835914 6738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:55.836493 6733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:55.837662 6733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:55.838835 6733 hybrid_clock.cc:648] HybridClock initialized: now 1790704075838818 us; error 28 us; skew 500 ppm
Sep 29 17:47:55 dist-test-slave-wbpt krb5kdc[6720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704075, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:55.841646 6733 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:55.842573 6733 webserver.cc:460] Webserver started at http://127.3.183.254:44845/ using document root <none> and password file <none>
I20260929 17:47:55.843101 6733 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:55.843159 6733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:55.843322 6733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:55.845361 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "45c1d074a8fc4f4bbc619b2135ebe364"
format_stamp: "Formatted at 2026-09-29 17:47:55 on dist-test-slave-wbpt"
server_key: "d6f6701b89948ed2b4256ec60df9d177"
server_key_iv: "2b8cb3029eae09a8315285ee063db2c5"
server_key_version: "encryptionkey@0"
I20260929 17:47:55.845860 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "45c1d074a8fc4f4bbc619b2135ebe364"
format_stamp: "Formatted at 2026-09-29 17:47:55 on dist-test-slave-wbpt"
server_key: "d6f6701b89948ed2b4256ec60df9d177"
server_key_iv: "2b8cb3029eae09a8315285ee063db2c5"
server_key_version: "encryptionkey@0"
I20260929 17:47:55.849392 6733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:47:55.851737 6748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:55.852955 6733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:55.853070 6733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "45c1d074a8fc4f4bbc619b2135ebe364"
format_stamp: "Formatted at 2026-09-29 17:47:55 on dist-test-slave-wbpt"
server_key: "d6f6701b89948ed2b4256ec60df9d177"
server_key_iv: "2b8cb3029eae09a8315285ee063db2c5"
server_key_version: "encryptionkey@0"
I20260929 17:47:55.853166 6733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:55.864519 6733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:55.867578 6733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:55.867821 6733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:55.874717 6733 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:43029
I20260929 17:47:55.874994 6800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:43029 every 8 connection(s)
I20260929 17:47:55.875710 6733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:55.878504 6801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:55.880524 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 6733
I20260929 17:47:55.880630 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:55.880861 3807 external_mini_cluster.cc:1428] Setting key fcdc5a31a3bea4f89e0f44ec27d3fb5d
I20260929 17:47:55.885226 6801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364: Bootstrap starting.
I20260929 17:47:55.887758 6801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:55.888538 6801 log.cc:824] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:47:55 dist-test-slave-wbpt krb5kdc[6720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:55.890390 6801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364: No bootstrap required, opened a new log
I20260929 17:47:55.894752 6801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43029 } }
I20260929 17:47:55.894995 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:55.882153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.3.183.254:43029)
0929 17:47:55.882587 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:55.882606 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:55.882649 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:47:55.883360 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:55.883385 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:55.883847 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:55.884224 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:55.884874 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:55.886270 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:55.887112 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:55.887800 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:55.887964 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:55.890203 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:55.890210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:55.890226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:55.890255 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:55.892126 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:55.892511 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:55.892518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:55.892524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:55.892673 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:55.892896 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:55.892899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:55.892901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:55.893393 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:55.893581 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:55.893752 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 17:47:55.894055 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260929 17:47:55.895021 6801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:55.895064 6801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1d074a8fc4f4bbc619b2135ebe364, State: Initialized, Role: FOLLOWER
I20260929 17:47:55.895529 6801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43029 } }
I20260929 17:47:55.895664 6801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:55.895727 6801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:55.895840 6801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:55.896921 6801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43029 } }
I20260929 17:47:55.897279 6801 leader_election.cc:302] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45c1d074a8fc4f4bbc619b2135ebe364; no voters:
I20260929 17:47:55.897627 6801 leader_election.cc:288] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:55.897704 6806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:55.897914 6806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [term 1 LEADER]: Becoming Leader. State: Replica: 45c1d074a8fc4f4bbc619b2135ebe364, State: Running, Role: LEADER
I20260929 17:47:55.897954 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:47:55.898232 6806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43029 } }
I20260929 17:47:55.898836 6801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:55.899405 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45c1d074a8fc4f4bbc619b2135ebe364. Latest consensus state: current_term: 1 leader_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43029 } } }
I20260929 17:47:55.899514 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:55.899941 6807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c1d074a8fc4f4bbc619b2135ebe364" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43029 } } }
I20260929 17:47:55.900117 6807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:55.899924 6812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:55.903283 6812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:55.909215 6812 catalog_manager.cc:1347] Generated new cluster ID: d1b5ed96a31f462082684c4f4358ef86
I20260929 17:47:55.909286 6812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:55.920737 6812 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:55.921010 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:55.915975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.3.183.254:43029)
0929 17:47:55.916080 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:55.916084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:55.916099 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:47:55.916158 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:55.916161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:55.916203 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:55.916270 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:55.916822 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:55.917595 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:55.918763 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:55.919093 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:55.919188 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:55.920048 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:55.920052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:55.920055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:47:55.920568 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:55.920596 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:47:55.920614 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:55.920691 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:55.920814 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 17:47:55.920876 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:47:55.921767 6812 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:55.928969 6812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45c1d074a8fc4f4bbc619b2135ebe364: Generated new TSK 0
I20260929 17:47:55.929625 6812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:47:55.936398 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 6733
2026-09-29T17:47:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1293 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:55 dist-test-slave-wbpt krb5kdc[6834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:55 dist-test-slave-wbpt krb5kdc[6834](info): set up 2 sockets
Sep 29 17:47:55 dist-test-slave-wbpt krb5kdc[6834](info): commencing operation
krb5kdc: starting...
W20260929 17:47:56.883543 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:56 dist-test-slave-wbpt krb5kdc[6834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:56.998486 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:33879
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:33879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:57.110277 6847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:57.110569 6847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:57.110613 6847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:57.114635 6847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:57.114701 6847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:57.114723 6847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:57.114741 6847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:57.114759 6847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:57.120158 6847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:33879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:33879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:57.121179 6847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:57.122153 6847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:57.130457 6853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:57.130592 6852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:57.130539 6855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:57.130878 6847 server_base.cc:1034] running on GCE node
I20260929 17:47:57.131451 6847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:57.132644 6847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:57.133817 6847 hybrid_clock.cc:648] HybridClock initialized: now 1790704077133801 us; error 31 us; skew 500 ppm
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:57.136509 6847 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:57.137447 6847 webserver.cc:460] Webserver started at http://127.3.183.254:37133/ using document root <none> and password file <none>
I20260929 17:47:57.137979 6847 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:57.138043 6847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:57.138265 6847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:57.140321 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "203c4c52725f43deaecc040313f06df6"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "01c76acd349fc07fe69d34ff1c6d5e9a"
server_key_iv: "382b09bc7245350bdc467708ab208713"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.140862 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "203c4c52725f43deaecc040313f06df6"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "01c76acd349fc07fe69d34ff1c6d5e9a"
server_key_iv: "382b09bc7245350bdc467708ab208713"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.144733 6847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:47:57.147135 6862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.148427 6847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:57.148541 6847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "203c4c52725f43deaecc040313f06df6"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "01c76acd349fc07fe69d34ff1c6d5e9a"
server_key_iv: "382b09bc7245350bdc467708ab208713"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.148648 6847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:57.191812 6847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:57.194918 6847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:57.195094 6847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:57.201618 6847 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:33879
I20260929 17:47:57.201872 6914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:33879 every 8 connection(s)
I20260929 17:47:57.202553 6847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:57.205097 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 6847
I20260929 17:47:57.205205 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:57.205389 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:57.205507 3807 external_mini_cluster.cc:1428] Setting key 2bed40e71eb5ea55ccb71ed5364774b0
I20260929 17:47:57.212556 6915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6: Bootstrap starting.
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:57.215380 6915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:57.216214 6915 log.cc:824] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:57.218261 6915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6: No bootstrap required, opened a new log
I20260929 17:47:57.220342 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.206789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.3.183.254:33879)
0929 17:47:57.207192 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:57.207209 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:57.207243 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:47:57.208070 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:57.208090 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:57.208429 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:57.208718 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:57.209777 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.211196 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.212428 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.212936 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.213050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.215928 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:57.215933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:57.215948 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:57.215979 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:57.217630 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.218048 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.218053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.218058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.218177 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.218448 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.218451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.218452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.218834 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:57.219004 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:57.219281 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 17:47:57.219558 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260929 17:47:57.221915 6915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c4c52725f43deaecc040313f06df6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33879 } }
I20260929 17:47:57.222155 6915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:57.222218 6915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203c4c52725f43deaecc040313f06df6, State: Initialized, Role: FOLLOWER
I20260929 17:47:57.222659 6915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c4c52725f43deaecc040313f06df6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33879 } }
I20260929 17:47:57.222792 6915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:57.222843 6915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:57.222920 6915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:57.223855 6915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c4c52725f43deaecc040313f06df6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33879 } }
I20260929 17:47:57.224157 6915 leader_election.cc:302] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203c4c52725f43deaecc040313f06df6; no voters:
I20260929 17:47:57.224424 6915 leader_election.cc:288] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:57.224515 6920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:57.224743 6920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [term 1 LEADER]: Becoming Leader. State: Replica: 203c4c52725f43deaecc040313f06df6, State: Running, Role: LEADER
I20260929 17:47:57.225070 6920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c4c52725f43deaecc040313f06df6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33879 } }
I20260929 17:47:57.225576 6915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:57.226666 6922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203c4c52725f43deaecc040313f06df6. Latest consensus state: current_term: 1 leader_uuid: "203c4c52725f43deaecc040313f06df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c4c52725f43deaecc040313f06df6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33879 } } }
I20260929 17:47:57.226665 6921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203c4c52725f43deaecc040313f06df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c4c52725f43deaecc040313f06df6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33879 } } }
I20260929 17:47:57.226799 6922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:57.226799 6921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:57.227285 6929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:57.229946 6929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:57.235239 6929 catalog_manager.cc:1347] Generated new cluster ID: 40f0bb1d9cbe472b84e8bdb94f9a4c59
I20260929 17:47:57.235306 6929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:57.261152 6929 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:57.262144 6929 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:57.267096 6929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 203c4c52725f43deaecc040313f06df6: Generated new TSK 0
I20260929 17:47:57.267757 6929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:47:57.298960 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33879
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:57.405380 6941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:57.405630 6941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:57.405670 6941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:57.409560 6941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:57.409607 6941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:57.409672 6941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:47:57.414747 6941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:57.415758 6941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:57.416785 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:57.425843 6946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:57.425828 6947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:57.425853 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:57.425966 6941 server_base.cc:1034] running on GCE node
I20260929 17:47:57.426474 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:57.427138 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:57.428289 6941 hybrid_clock.cc:648] HybridClock initialized: now 1790704077428264 us; error 34 us; skew 500 ppm
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:57.430866 6941 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:47:57.431694 6941 webserver.cc:460] Webserver started at http://127.3.183.193:32917/ using document root <none> and password file <none>
I20260929 17:47:57.432232 6941 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:57.432297 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:57.432463 6941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:57.434444 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "553d5eea37dc46abb29fa038fb943d1c"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "1d118b210f6aac687ed6486107246fbf"
server_key_iv: "68bdb2e086b66444df5336738ac231ff"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.434947 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "553d5eea37dc46abb29fa038fb943d1c"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "1d118b210f6aac687ed6486107246fbf"
server_key_iv: "68bdb2e086b66444df5336738ac231ff"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.438482 6941 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:47:57.440749 6956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.442040 6941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:57.442147 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "553d5eea37dc46abb29fa038fb943d1c"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "1d118b210f6aac687ed6486107246fbf"
server_key_iv: "68bdb2e086b66444df5336738ac231ff"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.442253 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:57.466570 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:57.469681 6941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:57.469852 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:57.470429 6941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:57.471230 6941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:57.471277 6941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.471321 6941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:57.471350 6941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.479553 6941 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:35983
I20260929 17:47:57.479574 7069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:35983 every 8 connection(s)
I20260929 17:47:57.480427 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:47:57.484817 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 6941
I20260929 17:47:57.484926 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:47:57.485162 3807 external_mini_cluster.cc:1428] Setting key 373ba10b2540864254fc624b2d0e4595
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:47:57.493602 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.482080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:43733 (local address 127.3.183.254:33879)
0929 17:47:57.482220 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:57.482224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:57.482987 (+ 763us) server_negotiation.cc:407] Connection header received
0929 17:47:57.484338 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:57.484341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:57.484385 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:57.484445 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:57.485782 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.486379 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.487460 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.487703 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.487811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.490010 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:57.490014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:57.490017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:57.490045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:57.491611 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.492142 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.492146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.492148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.492236 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.492538 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.492542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.492543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.492761 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:57.492863 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:57.493323 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 17:47:57.493436 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:47:57.494306 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.482397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33879 (local address 127.3.183.193:43733)
0929 17:47:57.482836 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:57.482886 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:57.484146 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:57.484620 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:57.484626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:57.484984 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:57.485490 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:57.485513 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.486488 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.486492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:57.487342 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:57.487350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.487848 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.487853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:57.488009 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.488510 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:57.488526 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:57.489857 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:57.491736 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:57.491741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:57.491759 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:57.492051 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:57.492319 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:57.492321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:57.492323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:57.492450 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:57.492857 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:57.492862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:57.493090 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 17:47:57.493229 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 17:47:57.493450 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260929 17:47:57.495430 7070 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33879
I20260929 17:47:57.495656 7070 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:47:57.496165 7070 heartbeater.cc:511] Master 127.3.183.254:33879 requested a full tablet report, sending...
I20260929 17:47:57.497663 6879 ts_manager.cc:194] Registered new tserver with Master: 553d5eea37dc46abb29fa038fb943d1c (127.3.183.193:35983)
I20260929 17:47:57.498734 6879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:43733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:47:57.508698 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33879
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:57.614810 7075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:57.615062 7075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:57.615118 7075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:57.619136 7075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:57.619197 7075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:57.619277 7075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:47:57.624472 7075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:57.625653 7075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:57.626663 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:57.635874 7083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:57.635938 7081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:57.635874 7080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:57.635892 7075 server_base.cc:1034] running on GCE node
I20260929 17:47:57.636487 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:57.637079 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:57.638253 7075 hybrid_clock.cc:648] HybridClock initialized: now 1790704077638231 us; error 32 us; skew 500 ppm
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:57.640760 7075 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:47:57.641618 7075 webserver.cc:460] Webserver started at http://127.3.183.194:43021/ using document root <none> and password file <none>
I20260929 17:47:57.642108 7075 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:57.642171 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:57.642354 7075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:57.644327 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "eda1f0ef4f2e44d79eeb070a1e0b9f12"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "577c6ed74880ebb99bed1855c1e29fb2"
server_key_iv: "e9d2da17f7ae69c00e41b466c40d17a0"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.644829 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "eda1f0ef4f2e44d79eeb070a1e0b9f12"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "577c6ed74880ebb99bed1855c1e29fb2"
server_key_iv: "e9d2da17f7ae69c00e41b466c40d17a0"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.648301 7075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:47:57.650873 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.652500 7075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:47:57.652614 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "eda1f0ef4f2e44d79eeb070a1e0b9f12"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "577c6ed74880ebb99bed1855c1e29fb2"
server_key_iv: "e9d2da17f7ae69c00e41b466c40d17a0"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.652726 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:57.674455 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:57.677662 7075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:57.677834 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:57.678385 7075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:57.679168 7075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:57.679221 7075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.679273 7075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:57.679297 7075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.687172 7075 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:45943
I20260929 17:47:57.687223 7203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:45943 every 8 connection(s)
I20260929 17:47:57.688078 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:47:57.696830 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7075
I20260929 17:47:57.696955 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:47:57.697288 3807 external_mini_cluster.cc:1428] Setting key 7d5644fd62aac193b1c7327febc8b598
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:57.701622 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.689821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:59773 (local address 127.3.183.254:33879)
0929 17:47:57.689957 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:57.689960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:57.690761 (+ 801us) server_negotiation.cc:407] Connection header received
0929 17:47:57.692080 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:57.692084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:57.692152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:57.692220 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:57.693320 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.693970 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.695178 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.695404 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.695504 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.697979 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:57.697983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:57.697986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:57.698009 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:57.699457 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.700004 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.700007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.700009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.700093 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.700487 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.700489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.700490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.700653 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:57.700724 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:57.701328 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 17:47:57.701472 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:47:57.702441 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.690194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33879 (local address 127.3.183.194:59773)
0929 17:47:57.690607 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:57.690663 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:57.691895 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:57.692385 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:57.692392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:57.692735 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:57.693193 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:57.693206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.694105 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.694108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:57.695013 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:57.695022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.695552 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.695556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:57.695714 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.696264 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:57.696279 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:57.697817 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:57.699589 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:57.699594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:57.699605 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:57.699908 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:57.700198 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:57.700202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:57.700204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:57.700393 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:57.700741 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:57.700747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:57.701060 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 17:47:57.701258 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 17:47:57.701565 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260929 17:47:57.703543 7204 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33879
I20260929 17:47:57.703837 7204 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:57.704414 7204 heartbeater.cc:511] Master 127.3.183.254:33879 requested a full tablet report, sending...
I20260929 17:47:57.705521 6879 ts_manager.cc:194] Registered new tserver with Master: eda1f0ef4f2e44d79eeb070a1e0b9f12 (127.3.183.194:45943)
I20260929 17:47:57.706095 6879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:59773
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:47:57.722337 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33879
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:57.828061 7209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:57.828294 7209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:57.828334 7209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:57.832485 7209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:57.832551 7209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:57.832609 7209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:47:57.837814 7209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:57.838778 7209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:57.839721 7209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:57.849378 7215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:57.849426 7209 server_base.cc:1034] running on GCE node
W20260929 17:47:57.849392 7214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:57.849385 7217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:57.850118 7209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:57.850780 7209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:57.851939 7209 hybrid_clock.cc:648] HybridClock initialized: now 1790704077851919 us; error 29 us; skew 500 ppm
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:57.854550 7209 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:47:57.855367 7209 webserver.cc:460] Webserver started at http://127.3.183.195:40399/ using document root <none> and password file <none>
I20260929 17:47:57.855891 7209 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:57.855953 7209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:57.856130 7209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:57.858240 7209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "f745f5382cf04064aa1a17424887087d"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "f4a0d917df2aaf072aded28b89ddddea"
server_key_iv: "c29edd94ce4b61a32bbe5793d31d74bd"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.858726 7209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "f745f5382cf04064aa1a17424887087d"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "f4a0d917df2aaf072aded28b89ddddea"
server_key_iv: "c29edd94ce4b61a32bbe5793d31d74bd"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.862178 7209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:47:57.864382 7224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.865499 7209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:57.865587 7209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "f745f5382cf04064aa1a17424887087d"
format_stamp: "Formatted at 2026-09-29 17:47:57 on dist-test-slave-wbpt"
server_key: "f4a0d917df2aaf072aded28b89ddddea"
server_key_iv: "c29edd94ce4b61a32bbe5793d31d74bd"
server_key_version: "encryptionkey@0"
I20260929 17:47:57.865661 7209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:57.888806 7209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:57.891340 7209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:57.891481 7209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:57.892007 7209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:57.892737 7209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:57.892781 7209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.892819 7209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:57.892839 7209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:57.900823 7209 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:33023
I20260929 17:47:57.900863 7337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:33023 every 8 connection(s)
I20260929 17:47:57.901633 7209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:47:57.908048 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7209
I20260929 17:47:57.908172 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:47:57.908469 3807 external_mini_cluster.cc:1428] Setting key de8af33df500852d00f4f8a1a3f7f7c0
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:57.914085 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.903259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:48931 (local address 127.3.183.254:33879)
0929 17:47:57.903370 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:57.903374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:57.904176 (+ 802us) server_negotiation.cc:407] Connection header received
0929 17:47:57.905451 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:57.905454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:57.905513 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:57.905583 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:57.906686 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.907208 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.908513 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.908747 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.908835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.911194 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:57.911197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:57.911199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:57.911221 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:57.912530 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.912943 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.912946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.912947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.913032 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.913267 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.913269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.913270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.913419 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:57.913488 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:57.913872 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 17:47:57.913959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:47:57.914776 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.903588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33879 (local address 127.3.183.195:48931)
0929 17:47:57.904026 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:57.904066 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:57.905251 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:57.905773 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:57.905779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:57.906105 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:57.906547 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:57.906559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.907317 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.907320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:57.908395 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:57.908406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.908892 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.908897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:57.909053 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.909621 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:57.909641 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:57.911056 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:57.912607 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:57.912611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:57.912620 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:57.912847 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:57.913098 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:57.913100 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:57.913101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:57.913208 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:57.913485 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:57.913490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:57.913698 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 17:47:57.913838 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 17:47:57.914054 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":142,"threads_started":1}
I20260929 17:47:57.915709 7338 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33879
I20260929 17:47:57.915925 7338 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:57.916318 7338 heartbeater.cc:511] Master 127.3.183.254:33879 requested a full tablet report, sending...
I20260929 17:47:57.917178 6879 ts_manager.cc:194] Registered new tserver with Master: f745f5382cf04064aa1a17424887087d (127.3.183.195:33023)
I20260929 17:47:57.917709 6879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:48931
I20260929 17:47:57.921723 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 17:47:57 dist-test-slave-wbpt krb5kdc[6834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704077, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:57.944619 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:57.937284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.3.183.254:33879)
0929 17:47:57.937402 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:57.937406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:57.937547 (+ 141us) server_negotiation.cc:407] Connection header received
0929 17:47:57.937651 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:57.937653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:57.937694 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:57.937762 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:57.938390 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.938942 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.939686 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:57.939931 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:57.940022 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:57.942099 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:57.942102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:57.942104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:57.942127 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:57.943392 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.943698 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.943701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.943702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.943767 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:57.943973 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:57.943975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:57.943976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:57.944182 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:57.944279 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:57.944415 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 17:47:57.944500 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 17:47:57.944970 6879 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47212
I20260929 17:47:57.945742 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 6941
I20260929 17:47:57.950834 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7075
I20260929 17:47:57.955531 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7209
I20260929 17:47:57.959771 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 6847
2026-09-29T17:47:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2024 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:47:58 dist-test-slave-wbpt krb5kdc[7350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:47:58 dist-test-slave-wbpt krb5kdc[7350](info): set up 2 sockets
Sep 29 17:47:58 dist-test-slave-wbpt krb5kdc[7350](info): commencing operation
krb5kdc: starting...
W20260929 17:47:58.923604 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:47:58 dist-test-slave-wbpt krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:47:59.037189 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46055
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:59.143226 7363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:59.143496 7363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:59.143551 7363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:59.147547 7363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:47:59.147619 7363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:59.147645 7363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:59.147665 7363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:47:59.147686 7363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:47:59.153033 7363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:46055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:59.154090 7363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:59.155072 7363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:59.163921 7369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:59.163973 7368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:59.163981 7363 server_base.cc:1034] running on GCE node
W20260929 17:47:59.163940 7371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:59.164758 7363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:59.165891 7363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:59.167045 7363 hybrid_clock.cc:648] HybridClock initialized: now 1790704079167022 us; error 39 us; skew 500 ppm
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:59.175980 7363 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:47:59.176923 7363 webserver.cc:460] Webserver started at http://127.3.183.254:38251/ using document root <none> and password file <none>
I20260929 17:47:59.177492 7363 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:59.177553 7363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:59.177762 7363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:59.179872 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "a1d5457284054a71af5d55ec5842f1d9"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "55127dbf82821cd0eadaf80add40df1e"
server_key_iv: "8795faace73b2a9ee7fbaa116395b900"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.180411 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "a1d5457284054a71af5d55ec5842f1d9"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "55127dbf82821cd0eadaf80add40df1e"
server_key_iv: "8795faace73b2a9ee7fbaa116395b900"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.184036 7363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:47:59.186297 7378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.187458 7363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:47:59.187553 7363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "a1d5457284054a71af5d55ec5842f1d9"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "55127dbf82821cd0eadaf80add40df1e"
server_key_iv: "8795faace73b2a9ee7fbaa116395b900"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.187670 7363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:59.206499 7363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:59.209723 7363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:59.209903 7363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:59.216877 7363 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:46055
I20260929 17:47:59.216933 7430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:46055 every 8 connection(s)
I20260929 17:47:59.217844 7363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:47:59.220665 7431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:47:59.223452 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7363
I20260929 17:47:59.223548 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:47:59.223829 3807 external_mini_cluster.cc:1428] Setting key 7f385795a8a836fac0f0d220f76af534
I20260929 17:47:59.226286 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9: Bootstrap starting.
I20260929 17:47:59.228691 7431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9: Neither blocks nor log segments found. Creating new log.
I20260929 17:47:59.229547 7431 log.cc:824] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:47:59.231678 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9: No bootstrap required, opened a new log
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:59.234855 7431 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d5457284054a71af5d55ec5842f1d9" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46055 } }
I20260929 17:47:59.235074 7431 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:47:59.235108 7431 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d5457284054a71af5d55ec5842f1d9, State: Initialized, Role: FOLLOWER
I20260929 17:47:59.235545 7431 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d5457284054a71af5d55ec5842f1d9" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46055 } }
I20260929 17:47:59.235675 7431 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:47:59.235745 7431 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:47:59.235886 7431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:47:59.236860 7431 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d5457284054a71af5d55ec5842f1d9" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46055 } }
I20260929 17:47:59.237253 7431 leader_election.cc:302] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1d5457284054a71af5d55ec5842f1d9; no voters:
I20260929 17:47:59.237547 7431 leader_election.cc:288] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:47:59.237651 7436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:47:59.237844 7436 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [term 1 LEADER]: Becoming Leader. State: Replica: a1d5457284054a71af5d55ec5842f1d9, State: Running, Role: LEADER
I20260929 17:47:59.238206 7436 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d5457284054a71af5d55ec5842f1d9" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46055 } }
I20260929 17:47:59.238339 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.225213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.3.183.254:46055)
0929 17:47:59.225590 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:59.225608 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:59.225644 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:47:59.226239 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:59.226255 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:59.226600 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:59.226867 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:59.227741 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.228769 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.229863 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.230459 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.230571 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.233214 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:59.233219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:59.233232 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:59.233258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:59.235292 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.235701 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.235712 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.235718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.235901 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.236189 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.236191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.236192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.236563 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:59.236714 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:59.237006 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 17:47:59.237343 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260929 17:47:59.238706 7431 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:47:59.239392 7436 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1d5457284054a71af5d55ec5842f1d9. Latest consensus state: current_term: 1 leader_uuid: "a1d5457284054a71af5d55ec5842f1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d5457284054a71af5d55ec5842f1d9" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46055 } } }
I20260929 17:47:59.239578 7436 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:59.240546 7445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:47:59.240973 7437 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1d5457284054a71af5d55ec5842f1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d5457284054a71af5d55ec5842f1d9" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46055 } } }
I20260929 17:47:59.242321 7437 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9 [sys.catalog]: This master's current role is: LEADER
I20260929 17:47:59.243461 7445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:47:59.249097 7445 catalog_manager.cc:1347] Generated new cluster ID: 8d595414d9d94334aeb8901e6ed1ed3d
I20260929 17:47:59.249166 7445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:47:59.262827 7445 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:47:59.263818 7445 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:47:59.275846 7445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1d5457284054a71af5d55ec5842f1d9: Generated new TSK 0
I20260929 17:47:59.276754 7445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:47:59.302453 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46055
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:59.409104 7457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:59.409340 7457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:59.409381 7457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:59.413348 7457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:59.413396 7457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:59.413465 7457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:47:59.418521 7457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:59.419466 7457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:59.420477 7457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:59.429606 7465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:59.429606 7462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:59.429620 7463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:59.429713 7457 server_base.cc:1034] running on GCE node
I20260929 17:47:59.430299 7457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:59.430922 7457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:59.432083 7457 hybrid_clock.cc:648] HybridClock initialized: now 1790704079432069 us; error 25 us; skew 500 ppm
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:59.434607 7457 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:47:59.435452 7457 webserver.cc:460] Webserver started at http://127.3.183.193:41053/ using document root <none> and password file <none>
I20260929 17:47:59.436022 7457 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:59.436088 7457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:59.436283 7457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:59.438303 7457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "d929c350bd5c45e9b12a5d9d60842330"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "221bc2c08a4e07afc0d5fb24797586ff"
server_key_iv: "38516b5502fa7ed2bfd34208ba499f3d"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.438812 7457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "d929c350bd5c45e9b12a5d9d60842330"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "221bc2c08a4e07afc0d5fb24797586ff"
server_key_iv: "38516b5502fa7ed2bfd34208ba499f3d"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.442493 7457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:47:59.444918 7472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.446162 7457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:47:59.446269 7457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "d929c350bd5c45e9b12a5d9d60842330"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "221bc2c08a4e07afc0d5fb24797586ff"
server_key_iv: "38516b5502fa7ed2bfd34208ba499f3d"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.446463 7457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:59.467018 7457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:59.469775 7457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:59.469944 7457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:59.470510 7457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:59.471287 7457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:59.471331 7457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.471382 7457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:59.471407 7457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.479552 7457 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:43563
I20260929 17:47:59.479575 7585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:43563 every 8 connection(s)
I20260929 17:47:59.480445 7457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:47:59.488116 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7457
I20260929 17:47:59.488231 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:47:59.488485 3807 external_mini_cluster.cc:1428] Setting key 0831e8eaa0642d85eaffd10e535facd5
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:59.493654 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.482189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:49293 (local address 127.3.183.254:46055)
0929 17:47:59.482310 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:59.482314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:59.483142 (+ 828us) server_negotiation.cc:407] Connection header received
0929 17:47:59.484510 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:59.484513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:59.484568 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:59.484640 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:59.485775 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.486301 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.487422 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.487654 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.487742 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.490274 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:59.490278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:59.490280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:59.490306 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:59.491713 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.492283 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.492285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.492286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.492353 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.492690 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.492692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.492693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.492841 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:59.492906 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:59.493446 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 17:47:59.493538 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:47:59.494868 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.482577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46055 (local address 127.3.183.193:49293)
0929 17:47:59.482998 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:59.483039 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:59.484312 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:59.484825 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:59.484831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:59.485171 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:59.485634 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:59.485647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.486414 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.486417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:59.487304 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:59.487313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.487818 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.487823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:59.487985 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.488532 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:59.488549 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:59.490119 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:59.491849 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:59.491855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:59.491870 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:59.492202 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:59.492426 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:59.492428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:59.492430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:59.492600 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:59.492922 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:59.492928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:59.493235 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 17:47:59.493415 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 17:47:59.493811 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260929 17:47:59.496008 7586 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46055
I20260929 17:47:59.496251 7586 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:59.496797 7586 heartbeater.cc:511] Master 127.3.183.254:46055 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:47:59.498284 7394 ts_manager.cc:194] Registered new tserver with Master: d929c350bd5c45e9b12a5d9d60842330 (127.3.183.193:43563)
I20260929 17:47:59.499405 7394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:49293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:47:59.514739 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46055
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:59.621685 7591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:59.621969 7591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:59.622033 7591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:59.626157 7591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:59.626225 7591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:59.626309 7591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:47:59.631547 7591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:59.632688 7591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:59.633690 7591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:59.643392 7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:59.643569 7591 server_base.cc:1034] running on GCE node
W20260929 17:47:59.643392 7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:59.643397 7599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:59.644117 7591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:59.644840 7591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:59.646004 7591 hybrid_clock.cc:648] HybridClock initialized: now 1790704079645988 us; error 32 us; skew 500 ppm
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:59.648635 7591 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:47:59.649533 7591 webserver.cc:460] Webserver started at http://127.3.183.194:41119/ using document root <none> and password file <none>
I20260929 17:47:59.650067 7591 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:59.650126 7591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:59.650292 7591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:59.652479 7591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "38d6e368390f435b9f11fcca24b8ef05"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "6685af60bca3cd84f03ffa82a143f472"
server_key_iv: "19d79ac2b5344ee51cc55a2358576f71"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.653014 7591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "38d6e368390f435b9f11fcca24b8ef05"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "6685af60bca3cd84f03ffa82a143f472"
server_key_iv: "19d79ac2b5344ee51cc55a2358576f71"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.656836 7591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:47:59.659333 7606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.660686 7591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:47:59.660787 7591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "38d6e368390f435b9f11fcca24b8ef05"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "6685af60bca3cd84f03ffa82a143f472"
server_key_iv: "19d79ac2b5344ee51cc55a2358576f71"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.660897 7591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:59.670334 7591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:59.673318 7591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:59.673483 7591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:59.674029 7591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:59.674810 7591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:59.674853 7591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.674896 7591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:59.674923 7591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.683454 7591 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:41489
I20260929 17:47:59.683475 7719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:41489 every 8 connection(s)
I20260929 17:47:59.684332 7591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:47:59.690377 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7591
I20260929 17:47:59.690529 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:47:59.690807 3807 external_mini_cluster.cc:1428] Setting key 4caf854a9689e7aeda15d0a88b69de58
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:59.698216 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.686150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:43263 (local address 127.3.183.254:46055)
0929 17:47:59.686269 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:59.686273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:59.687109 (+ 836us) server_negotiation.cc:407] Connection header received
0929 17:47:59.688468 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:59.688471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:59.688529 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:59.688604 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:59.689832 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.690420 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.691752 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.692040 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.692152 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.694742 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:59.694745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:59.694747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:59.694770 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:59.696110 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.696725 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.696728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.696729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.696823 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.697139 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.697142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.697144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.697337 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:59.697424 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:59.697920 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 17:47:59.698055 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:47:59.699165 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.686485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46055 (local address 127.3.183.194:43263)
0929 17:47:59.686952 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:59.686995 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:59.688263 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:59.688792 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:59.688799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:59.689163 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:59.689695 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:59.689709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.690556 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.690560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:59.691644 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:59.691657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.692202 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.692206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:59.692406 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.693001 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:59.693021 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:59.694589 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:59.696283 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:59.696290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:59.696308 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:59.696626 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:59.696909 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:59.696912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:59.696914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:59.697053 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:59.697447 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:59.697453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:59.697693 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 17:47:59.697860 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 17:47:59.698200 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:47:59.700443 7720 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46055
I20260929 17:47:59.700693 7720 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:59.701216 7720 heartbeater.cc:511] Master 127.3.183.254:46055 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:47:59.702445 7394 ts_manager.cc:194] Registered new tserver with Master: 38d6e368390f435b9f11fcca24b8ef05 (127.3.183.194:41489)
I20260929 17:47:59.703017 7394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:43263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:47:59.719260 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46055
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:47:59.828157 7725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:47:59.828403 7725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:47:59.828447 7725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:47:59.832469 7725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:47:59.832523 7725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:47:59.832581 7725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:47:59.837927 7725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:47:59.838948 7725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:47:59.840068 7725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:47:59.849323 7733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:59.849409 7725 server_base.cc:1034] running on GCE node
W20260929 17:47:59.849310 7731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:47:59.849546 7730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:47:59.849954 7725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:47:59.850548 7725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:47:59.851711 7725 hybrid_clock.cc:648] HybridClock initialized: now 1790704079851681 us; error 40 us; skew 500 ppm
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:47:59.854707 7725 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:47:59.855556 7725 webserver.cc:460] Webserver started at http://127.3.183.195:38429/ using document root <none> and password file <none>
I20260929 17:47:59.856122 7725 fs_manager.cc:359] Metadata directory not provided
I20260929 17:47:59.856184 7725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:47:59.856365 7725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:47:59.858382 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "4d4c1f9a6ede4ccfaaccb6ff5b3ed1d2"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "6eeb94daa94608e874d52fb0b8446a97"
server_key_iv: "e4270e93bd59484fc7d8ac98f6dc898d"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.858930 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "4d4c1f9a6ede4ccfaaccb6ff5b3ed1d2"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "6eeb94daa94608e874d52fb0b8446a97"
server_key_iv: "e4270e93bd59484fc7d8ac98f6dc898d"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.862680 7725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 17:47:59.865181 7740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.866364 7725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:47:59.866493 7725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "4d4c1f9a6ede4ccfaaccb6ff5b3ed1d2"
format_stamp: "Formatted at 2026-09-29 17:47:59 on dist-test-slave-wbpt"
server_key: "6eeb94daa94608e874d52fb0b8446a97"
server_key_iv: "e4270e93bd59484fc7d8ac98f6dc898d"
server_key_version: "encryptionkey@0"
I20260929 17:47:59.866578 7725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:47:59.882954 7725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:47:59.885922 7725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:47:59.886073 7725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:47:59.886606 7725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:47:59.887387 7725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:47:59.887444 7725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.887483 7725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:47:59.887506 7725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:47:59.896042 7725 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:33565
I20260929 17:47:59.896093 7853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:33565 every 8 connection(s)
I20260929 17:47:59.896886 7725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:47:59.905423 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7725
I20260929 17:47:59.905527 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:47:59.905804 3807 external_mini_cluster.cc:1428] Setting key 44c1bef0836c22c25eff059a926e40bd
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:59.910311 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.898724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:35969 (local address 127.3.183.254:46055)
0929 17:47:59.898860 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:59.898864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:59.899715 (+ 851us) server_negotiation.cc:407] Connection header received
0929 17:47:59.901087 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:59.901090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:59.901138 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:59.901209 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:59.902470 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.903045 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.904231 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.904460 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.904546 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.907191 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:59.907194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:59.907196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:59.907221 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:59.908627 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.909072 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.909074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.909075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.909152 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.909413 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.909415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.909416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.909592 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:59.909662 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:59.910097 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 17:47:59.910193 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:47:59.911164 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.899148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46055 (local address 127.3.183.195:35969)
0929 17:47:59.899574 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 17:47:59.899616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:47:59.900881 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:47:59.901413 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:47:59.901420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:47:59.901820 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:47:59.902306 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:59.902320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.903174 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.903177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:59.904117 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:47:59.904129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.904590 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.904592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:47:59.904749 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.905346 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:47:59.905363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:47:59.907024 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:47:59.908725 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:59.908730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:59.908743 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:59.908989 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:59.909220 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:47:59.909222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:47:59.909224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:47:59.909349 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:47:59.909661 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:47:59.909665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:47:59.909908 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 17:47:59.910064 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 17:47:59.910334 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260929 17:47:59.912181 7854 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46055
I20260929 17:47:59.912415 7854 heartbeater.cc:464] Registering TS with master...
I20260929 17:47:59.912878 7854 heartbeater.cc:511] Master 127.3.183.254:46055 requested a full tablet report, sending...
I20260929 17:47:59.913823 7394 ts_manager.cc:194] Registered new tserver with Master: 4d4c1f9a6ede4ccfaaccb6ff5b3ed1d2 (127.3.183.195:33565)
I20260929 17:47:59.914357 7394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:35969
I20260929 17:47:59.919765 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:47:59.944816 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.935418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.3.183.193:43563)
0929 17:47:59.935713 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:59.935725 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:59.935747 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:47:59.935832 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:59.935836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:59.936033 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:59.936156 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:59.936922 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.937834 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.938756 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.939160 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.939257 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.941558 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:59.941567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:59.941589 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:59.941613 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:59.943083 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.943477 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.943483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.943485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.943561 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.943828 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.943831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.943833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.944072 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:59.944264 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:59.944501 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 17:47:59.944653 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
Sep 29 17:47:59 dist-test-slave-wbpt krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:47:59.955132 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:47:59.947432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.3.183.254:46055)
0929 17:47:59.947598 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 17:47:59.947604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:47:59.947670 (+ 66us) server_negotiation.cc:407] Connection header received
0929 17:47:59.947759 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:47:59.947761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:47:59.947834 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:47:59.947915 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:47:59.948416 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.948943 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.949678 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:47:59.949891 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:47:59.949972 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:47:59.952251 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:47:59.952254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:47:59.952256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:47:59.952279 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:47:59.953648 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.954050 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.954053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.954054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.954136 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:47:59.954339 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:47:59.954342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:47:59.954343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:47:59.954568 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:47:59.954671 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:47:59.954861 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 17:47:59.954970 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 17:47:59.955515 7394 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196
I20260929 17:47:59.956120 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7457
I20260929 17:47:59.960832 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7591
I20260929 17:47:59.965385 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7725
I20260929 17:47:59.969591 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7363
2026-09-29T17:47:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2010 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:00 dist-test-slave-wbpt krb5kdc[7866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:00 dist-test-slave-wbpt krb5kdc[7866](info): set up 2 sockets
Sep 29 17:48:00 dist-test-slave-wbpt krb5kdc[7866](info): commencing operation
krb5kdc: starting...
W20260929 17:48:00.939626 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:48:01.059950 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:37787
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:37787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:01.167301 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:01.167570 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:01.167629 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:01.171569 7879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:01.171627 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:01.171654 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:01.171674 7879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:01.171694 7879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:01.176837 7879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:37787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:37787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:01.177846 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:01.178807 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:01.186683 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:01.186656 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:01.186843 7879 server_base.cc:1034] running on GCE node
W20260929 17:48:01.186662 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:01.187434 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:01.188501 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:01.189664 7879 hybrid_clock.cc:648] HybridClock initialized: now 1790704081189648 us; error 29 us; skew 500 ppm
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:01.192212 7879 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:48:01.192966 7879 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:01.193045 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:01.193252 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:01.195242 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "84b613710692792bfae4cde3a4ff0489"
server_key_iv: "1f134c4c3fa48c2a11e9fc288d2ee02e"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.195747 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "84b613710692792bfae4cde3a4ff0489"
server_key_iv: "1f134c4c3fa48c2a11e9fc288d2ee02e"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.199378 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:48:01.201779 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.203032 7879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:01.203169 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "84b613710692792bfae4cde3a4ff0489"
server_key_iv: "1f134c4c3fa48c2a11e9fc288d2ee02e"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.203274 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:01.225337 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:01.228521 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:01.228719 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:01.235347 7879 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:37787
I20260929 17:48:01.235580 7945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:37787 every 8 connection(s)
I20260929 17:48:01.236042 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:01.238776 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:01.244299 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4: Bootstrap starting.
I20260929 17:48:01.246250 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7879
I20260929 17:48:01.246387 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:01.246650 3807 external_mini_cluster.cc:1428] Setting key ae9c395b2cb85301d0cee7c98ed52ea3
I20260929 17:48:01.246990 7946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:01.247923 7946 log.cc:824] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:01.250168 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4: No bootstrap required, opened a new log
I20260929 17:48:01.254449 7946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37787 } }
I20260929 17:48:01.254698 7946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:01.254736 7946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e773d7939a4e6ea9ca6a2a65f95ff4, State: Initialized, Role: FOLLOWER
I20260929 17:48:01.255198 7946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37787 } }
I20260929 17:48:01.255330 7946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:01.255395 7946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:01.255489 7946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:01.256515 7946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37787 } }
I20260929 17:48:01.256903 7946 leader_election.cc:302] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e773d7939a4e6ea9ca6a2a65f95ff4; no voters:
I20260929 17:48:01.257290 7946 leader_election.cc:288] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:01.257397 7951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:01.257601 7951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [term 1 LEADER]: Becoming Leader. State: Replica: 60e773d7939a4e6ea9ca6a2a65f95ff4, State: Running, Role: LEADER
I20260929 17:48:01.257910 7951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37787 } }
I20260929 17:48:01.258701 7946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:01.259011 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e773d7939a4e6ea9ca6a2a65f95ff4. Latest consensus state: current_term: 1 leader_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37787 } } }
I20260929 17:48:01.259141 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:01.259513 7955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:01.259545 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e773d7939a4e6ea9ca6a2a65f95ff4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37787 } } }
I20260929 17:48:01.259728 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:01.262205 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.247872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.3.183.254:37787)
0929 17:48:01.248345 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.248366 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.248401 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:48:01.249183 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.249207 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.249614 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.249981 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:01.250702 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.252231 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.253104 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.253874 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.254036 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.256466 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:01.256473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:01.256491 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:01.256524 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:01.258521 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.259059 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.259065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.259071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.259212 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.259522 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.259526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.259528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.260058 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:01.260812 (+ 754us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.260979 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.261269 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":172,"threads_started":1}
I20260929 17:48:01.262841 7955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:01.267887 7955 catalog_manager.cc:1347] Generated new cluster ID: 280a4ba02fcf4697be1fd96ac1013036
I20260929 17:48:01.267952 7955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:01.282727 7955 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:01.283739 7955 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:01.287686 7955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4: Generated new TSK 0
I20260929 17:48:01.288344 7955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:48:01.319298 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:37787
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:01.435391 7972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:01.435671 7972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:01.435738 7972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:01.439893 7972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:01.439956 7972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:01.440040 7972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:01.445202 7972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:37787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:01.446223 7972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:01.447221 7972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:01.456449 7980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:01.456552 7977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:01.456447 7978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:01.456759 7972 server_base.cc:1034] running on GCE node
I20260929 17:48:01.457213 7972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:01.457815 7972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:01.458974 7972 hybrid_clock.cc:648] HybridClock initialized: now 1790704081458944 us; error 42 us; skew 500 ppm
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:01.461792 7972 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:48:01.462463 7972 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:01.462522 7972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:01.462687 7972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:01.464665 7972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "cd392ede5942406a8299f960e8409a92"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "fcb14d0601b43d3e8aea6c34678d94b8"
server_key_iv: "363db89c0f23c59c20f0e3916c0b4a05"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.465153 7972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "cd392ede5942406a8299f960e8409a92"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "fcb14d0601b43d3e8aea6c34678d94b8"
server_key_iv: "363db89c0f23c59c20f0e3916c0b4a05"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.468681 7972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:01.471064 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.472347 7972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:01.472442 7972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "cd392ede5942406a8299f960e8409a92"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "fcb14d0601b43d3e8aea6c34678d94b8"
server_key_iv: "363db89c0f23c59c20f0e3916c0b4a05"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.472541 7972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:01.511130 7972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:01.514264 7972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:01.514430 7972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:01.515007 7972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:01.515820 7972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:01.515864 7972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.515902 7972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:01.515920 7972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.523926 7972 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:35155
I20260929 17:48:01.523977 8099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:35155 every 8 connection(s)
I20260929 17:48:01.524488 7972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:01.526315 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 7972
I20260929 17:48:01.526429 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:01.526718 3807 external_mini_cluster.cc:1428] Setting key d69b672c2b9e1714a0c0461e4da7be92
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:48:01.537940 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.526252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:59015 (local address 127.3.183.254:37787)
0929 17:48:01.526377 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.526382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.527416 (+ 1034us) server_negotiation.cc:407] Connection header received
0929 17:48:01.528871 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.528874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.528926 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.528997 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:01.530157 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.530709 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.531832 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.532076 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.532164 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.534572 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:01.534575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:01.534577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:01.534601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:01.535948 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.536442 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.536445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.536446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.536574 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.536843 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.536845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.536847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.537074 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:01.537148 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.537633 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.537762 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:48:01.538887 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.526730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:37787 (local address 127.3.183.193:59015)
0929 17:48:01.527267 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:01.527323 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:01.528667 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:01.529158 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:01.529165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:01.529524 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:01.530018 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.530031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.530829 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.530832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.531712 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.531721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.532207 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.532211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.532365 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.532889 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:01.532905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:01.534382 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:01.536044 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:01.536049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:01.536061 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:01.536345 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:01.536655 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:01.536657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:01.536658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:01.536781 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:01.537178 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:01.537183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:01.537407 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 17:48:01.537590 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 17:48:01.537891 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260929 17:48:01.540174 8100 heartbeater.cc:347] Connected to a master server at 127.3.183.254:37787
I20260929 17:48:01.540413 8100 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:01.540881 8100 heartbeater.cc:511] Master 127.3.183.254:37787 requested a full tablet report, sending...
I20260929 17:48:01.542420 7909 ts_manager.cc:194] Registered new tserver with Master: cd392ede5942406a8299f960e8409a92 (127.3.183.193:35155)
I20260929 17:48:01.543969 7909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:59015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:48:01.554241 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:37787
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:01.661849 8105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:01.662112 8105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:01.662158 8105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:01.666086 8105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:01.666141 8105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:01.666201 8105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:01.671263 8105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:37787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:01.672276 8105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:01.673226 8105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:48:01.682531 8105 server_base.cc:1034] running on GCE node
W20260929 17:48:01.682472 8110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:01.682510 8113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:01.682467 8111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:01.683120 8105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:01.683734 8105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:01.684911 8105 hybrid_clock.cc:648] HybridClock initialized: now 1790704081684887 us; error 35 us; skew 500 ppm
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:01.687610 8105 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:48:01.688337 8105 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:01.688398 8105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:01.688599 8105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:01.690615 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "1bdc2c2965ba35a52fb8b31916ba359a"
server_key_iv: "67b4cc5425ba2b07bf8f00e556c75254"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.691159 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "1bdc2c2965ba35a52fb8b31916ba359a"
server_key_iv: "67b4cc5425ba2b07bf8f00e556c75254"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.694942 8105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:48:01.697481 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.698761 8105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:01.698863 8105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "1bdc2c2965ba35a52fb8b31916ba359a"
server_key_iv: "67b4cc5425ba2b07bf8f00e556c75254"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.698975 8105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:01.720660 8105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:01.723650 8105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:01.723860 8105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:01.724465 8105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:01.725308 8105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:01.725395 8105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.725472 8105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:01.725497 8105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.734272 8105 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:46711
I20260929 17:48:01.734292 8232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:46711 every 8 connection(s)
I20260929 17:48:01.734879 8105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:01.739956 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 8105
I20260929 17:48:01.740090 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:01.740401 3807 external_mini_cluster.cc:1428] Setting key 31f606034f901f8f059299333c901fb0
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:01.749218 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.736642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:38147 (local address 127.3.183.254:37787)
0929 17:48:01.736769 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.736773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.737672 (+ 899us) server_negotiation.cc:407] Connection header received
0929 17:48:01.738927 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.738930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.738980 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.739053 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:01.740424 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.741018 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.742382 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.742748 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.742852 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.745542 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:01.745545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:01.745547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:01.745569 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:01.746859 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.747474 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.747476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.747478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.747557 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.747947 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.747951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.747952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.748137 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:01.748255 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.748976 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.749092 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:48:01.750226 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.737024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:37787 (local address 127.3.183.194:38147)
0929 17:48:01.737494 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:01.737538 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:01.738729 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:01.739219 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:01.739226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:01.739581 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:01.740274 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.740295 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.741142 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.741146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.742263 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.742272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.742876 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.742878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.743017 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.743616 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:01.743644 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:01.745381 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:01.746966 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:01.746973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:01.746989 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:01.747334 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:01.747653 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:01.747656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:01.747658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:01.747859 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:01.748273 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:01.748280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:01.748593 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 17:48:01.748842 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 17:48:01.749196 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260929 17:48:01.751296 8233 heartbeater.cc:347] Connected to a master server at 127.3.183.254:37787
I20260929 17:48:01.751544 8233 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:01.752048 8233 heartbeater.cc:511] Master 127.3.183.254:37787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:48:01.753047 7909 ts_manager.cc:194] Registered new tserver with Master: 129b7c9f5c0743dd9eadbf1256c5f2b6 (127.3.183.194:46711)
I20260929 17:48:01.753659 7909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:38147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:48:01.766669 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:37787
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:01.873974 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:01.874248 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:01.874291 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:01.878315 8238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:01.878394 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:01.878479 8238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:01.883683 8238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:37787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:01.884724 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:01.885680 8238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:01.894907 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:01.894928 8244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:01.894956 8238 server_base.cc:1034] running on GCE node
W20260929 17:48:01.895107 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:01.895545 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:01.896198 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:01.897373 8238 hybrid_clock.cc:648] HybridClock initialized: now 1790704081897356 us; error 29 us; skew 500 ppm
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:01.899986 8238 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:48:01.900657 8238 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:01.900717 8238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:01.900877 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:01.902891 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "1414a591f2cf4a429331a47e060ec6be"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "780574477170309f5f01f36d65ebbff3"
server_key_iv: "605529a92c04bcb49ea071f064caa10d"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.903411 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "1414a591f2cf4a429331a47e060ec6be"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "780574477170309f5f01f36d65ebbff3"
server_key_iv: "605529a92c04bcb49ea071f064caa10d"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.906872 8238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:48:01.909183 8252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.910348 8238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:01.910452 8238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "1414a591f2cf4a429331a47e060ec6be"
format_stamp: "Formatted at 2026-09-29 17:48:01 on dist-test-slave-wbpt"
server_key: "780574477170309f5f01f36d65ebbff3"
server_key_iv: "605529a92c04bcb49ea071f064caa10d"
server_key_version: "encryptionkey@0"
I20260929 17:48:01.910589 8238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:01.927187 8238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:01.929878 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:01.930047 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:01.930596 8238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:01.931387 8238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:01.931434 8238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.931506 8238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:01.931537 8238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:01.940114 8238 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:32795
I20260929 17:48:01.940151 8365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:32795 every 8 connection(s)
I20260929 17:48:01.940680 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:01.942373 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 8238
I20260929 17:48:01.942497 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:01.942794 3807 external_mini_cluster.cc:1428] Setting key 522f5e6d5b5a1ab5752bd9474fc195d9
Sep 29 17:48:01 dist-test-slave-wbpt krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:01.953631 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.942456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:60079 (local address 127.3.183.254:37787)
0929 17:48:01.942594 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.942599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.943745 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 17:48:01.945108 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.945111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.945170 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.945233 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:01.946382 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.946884 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.948074 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.948297 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.948387 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.950670 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:01.950673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:01.950675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:01.950698 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:01.951919 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.952378 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.952380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.952382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.952449 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.952701 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.952703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.952704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.952854 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:01.952919 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.953425 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.953511 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:48:01.954485 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.942998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:37787 (local address 127.3.183.195:60079)
0929 17:48:01.943519 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:01.943568 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:01.944926 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:01.945381 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:01.945388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:01.945779 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:01.946278 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.946291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.946972 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.946974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.947893 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.947902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.948431 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.948433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.948601 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.949114 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:01.949133 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:01.950520 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:01.952033 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:01.952038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:01.952049 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:01.952292 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:01.952511 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:01.952513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:01.952514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:01.952637 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:01.953004 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:01.953009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:01.953236 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 17:48:01.953407 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 17:48:01.953659 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
I20260929 17:48:01.955626 8366 heartbeater.cc:347] Connected to a master server at 127.3.183.254:37787
I20260929 17:48:01.955878 8366 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:01.956313 8366 heartbeater.cc:511] Master 127.3.183.254:37787 requested a full tablet report, sending...
I20260929 17:48:01.957276 7909 ts_manager.cc:194] Registered new tserver with Master: 1414a591f2cf4a429331a47e060ec6be (127.3.183.195:32795)
I20260929 17:48:01.957819 7909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:60079
I20260929 17:48:01.968034 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:48:01.977099 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.969913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.3.183.254:37787)
0929 17:48:01.970100 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.970105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.970248 (+ 143us) server_negotiation.cc:407] Connection header received
0929 17:48:01.970373 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.970376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.970451 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.970582 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:01.971216 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.971717 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.972504 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.972722 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.972798 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.974114 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:01.974116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:01.974119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:01.974140 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:01.975435 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.975902 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.975906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.975908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.975984 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:01.976228 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:01.976231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:01.976232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:01.976452 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:01.976565 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.976790 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.976920 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:48:01.979878 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:01.982820 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:01.997082 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.991446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:46711 (local address 127.0.0.1:46036)
0929 17:48:01.992198 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:01.992211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:01.992288 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:01.993104 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:01.993110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:01.993134 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:01.993285 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.993293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.994392 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.994395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.995604 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.995613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.996526 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.996530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.996645 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.996684 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 17:48:01.996809 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 17:48:01.996925 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":198,"thread_start_us":123,"threads_started":1}
I20260929 17:48:01.997095 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.991688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.3.183.194:46711)
0929 17:48:01.992614 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.992625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.992648 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:48:01.992726 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.992731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.992945 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.993105 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:01.993422 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.994272 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.995717 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.996367 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.996492 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.996639 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.996843 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.996942 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":132,"threads_started":1}
I20260929 17:48:01.997717 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.992270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:32795 (local address 127.0.0.1:42550)
0929 17:48:01.992698 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:01.992710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:01.992792 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:01.993362 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:01.993365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:01.993383 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:01.993479 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.993486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.994875 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.994878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.996139 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.996149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.997325 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.997329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.997423 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.997443 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:48:01.997509 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:48:01.997578 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":23,"thread_start_us":85,"threads_started":1}
I20260929 17:48:01.997799 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.992588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.3.183.195:32795)
0929 17:48:01.992947 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.992958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.992978 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:48:01.993058 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.993063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.993241 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.993371 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:01.993592 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.994734 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.996264 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.997001 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.997098 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.997262 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.997555 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.997650 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":169,"threads_started":1}
I20260929 17:48:01.998193 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.991556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.3.183.193:35155)
0929 17:48:01.992985 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:01.992990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:01.993009 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:48:01.993081 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:01.993086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:01.993273 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:01.993405 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:01.993783 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.994855 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.996298 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.997191 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.997295 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.997419 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:01.997944 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 17:48:01.998054 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":72,"threads_started":1}
I20260929 17:48:01.997854 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:01.991415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:35155 (local address 127.0.0.1:51304)
0929 17:48:01.992068 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:01.992119 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:01.992248 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:01.993558 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:01.993561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:01.993576 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:01.993663 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.993668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.994976 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.994979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.996212 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:01.996220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:01.997476 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:01.997480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:01.997578 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:01.997600 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:48:01.997665 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:48:01.997733 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260929 17:48:02.001050 8300 tablet_service.cc:1467] Processing CreateTablet for tablet ede6edaf923c49849070ce1a49f20fa4 (DEFAULT_TABLE table=test-table [id=e66c6a7aecd44fb2964478d880235e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:02.001050 8167 tablet_service.cc:1467] Processing CreateTablet for tablet ede6edaf923c49849070ce1a49f20fa4 (DEFAULT_TABLE table=test-table [id=e66c6a7aecd44fb2964478d880235e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:02.001420 8034 tablet_service.cc:1467] Processing CreateTablet for tablet ede6edaf923c49849070ce1a49f20fa4 (DEFAULT_TABLE table=test-table [id=e66c6a7aecd44fb2964478d880235e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:02.002282 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede6edaf923c49849070ce1a49f20fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:02.002287 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede6edaf923c49849070ce1a49f20fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:02.002466 8034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede6edaf923c49849070ce1a49f20fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:02.009120 8383 tablet_bootstrap.cc:492] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92: Bootstrap starting.
I20260929 17:48:02.010362 8385 tablet_bootstrap.cc:492] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Bootstrap starting.
I20260929 17:48:02.011459 8383 tablet_bootstrap.cc:654] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:02.012163 8384 tablet_bootstrap.cc:492] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Bootstrap starting.
I20260929 17:48:02.012288 8385 tablet_bootstrap.cc:654] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:02.012908 8383 log.cc:824] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:02.013583 8385 log.cc:824] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:02.014776 8384 tablet_bootstrap.cc:654] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:02.015113 8383 tablet_bootstrap.cc:492] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92: No bootstrap required, opened a new log
I20260929 17:48:02.015331 8383 ts_tablet_manager.cc:1397] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 17:48:02.015579 8384 log.cc:824] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:02.015803 8385 tablet_bootstrap.cc:492] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: No bootstrap required, opened a new log
I20260929 17:48:02.015990 8385 ts_tablet_manager.cc:1397] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:48:02.017617 8384 tablet_bootstrap.cc:492] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: No bootstrap required, opened a new log
I20260929 17:48:02.017809 8384 ts_tablet_manager.cc:1397] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:48:02.018730 8383 raft_consensus.cc:348] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.018962 8383 raft_consensus.cc:374] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:02.019034 8383 raft_consensus.cc:729] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd392ede5942406a8299f960e8409a92, State: Initialized, Role: FOLLOWER
I20260929 17:48:02.019490 8383 consensus_queue.cc:260] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.019665 8385 raft_consensus.cc:348] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.020013 8385 raft_consensus.cc:374] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:02.020113 8385 raft_consensus.cc:729] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1414a591f2cf4a429331a47e060ec6be, State: Initialized, Role: FOLLOWER
I20260929 17:48:02.020628 8385 consensus_queue.cc:260] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.020761 8383 ts_tablet_manager.cc:1428] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 17:48:02.021183 8100 heartbeater.cc:503] Master 127.3.183.254:37787 was elected leader, sending a full tablet report...
I20260929 17:48:02.021390 8366 heartbeater.cc:503] Master 127.3.183.254:37787 was elected leader, sending a full tablet report...
I20260929 17:48:02.021454 8384 raft_consensus.cc:348] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.021703 8384 raft_consensus.cc:374] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:02.021790 8384 raft_consensus.cc:729] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129b7c9f5c0743dd9eadbf1256c5f2b6, State: Initialized, Role: FOLLOWER
I20260929 17:48:02.022207 8384 consensus_queue.cc:260] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.022749 8385 ts_tablet_manager.cc:1428] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 17:48:02.023128 8384 ts_tablet_manager.cc:1428] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:48:02.024066 8233 heartbeater.cc:503] Master 127.3.183.254:37787 was elected leader, sending a full tablet report...
W20260929 17:48:02.025496 8101 tablet.cc:2369] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:02.152122 8389 raft_consensus.cc:482] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:02.152328 8389 raft_consensus.cc:504] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.153479 8389 leader_election.cc:288] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129b7c9f5c0743dd9eadbf1256c5f2b6 (127.3.183.194:46711), 1414a591f2cf4a429331a47e060ec6be (127.3.183.195:32795)
I20260929 17:48:02.157179 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.153714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:46711 (local address 127.3.183.193:43555)
0929 17:48:02.153972 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:02.153997 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:02.154106 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:02.154475 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:02.154477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:02.154497 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:02.154598 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:02.154602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.155331 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.155333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:02.156152 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:02.156161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.156769 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.156773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:02.156866 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:02.156884 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 17:48:02.156955 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:48:02.157019 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260929 17:48:02.157152 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.153827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:43555 (local address 127.3.183.194:46711)
0929 17:48:02.153947 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.153951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.154092 (+ 141us) server_negotiation.cc:407] Connection header received
0929 17:48:02.154215 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.154218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.154279 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.154368 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:02.154715 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.155222 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.156268 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.156667 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.156769 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:02.156822 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.156975 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.157042 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:48:02.157419 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.154332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:53985 (local address 127.3.183.195:32795)
0929 17:48:02.154436 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.154440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.154531 (+ 91us) server_negotiation.cc:407] Connection header received
0929 17:48:02.154648 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.154663 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.154703 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.154762 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:02.155014 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.155532 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.156562 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.156942 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.157042 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:02.157089 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.157253 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.157314 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 17:48:02.157435 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.154089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:32795 (local address 127.3.183.193:53985)
0929 17:48:02.154452 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:02.154465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:02.154563 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:02.154787 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:02.154789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:02.154804 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:02.154913 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:02.154918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.155636 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.155639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:02.156451 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:02.156461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.157042 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.157046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:02.157137 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:02.157170 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 17:48:02.157231 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:48:02.157295 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":133,"threads_started":1}
I20260929 17:48:02.158034 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede6edaf923c49849070ce1a49f20fa4" candidate_uuid: "cd392ede5942406a8299f960e8409a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1414a591f2cf4a429331a47e060ec6be" is_pre_election: true
I20260929 17:48:02.158034 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede6edaf923c49849070ce1a49f20fa4" candidate_uuid: "cd392ede5942406a8299f960e8409a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" is_pre_election: true
I20260929 17:48:02.158346 8187 raft_consensus.cc:2454] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd392ede5942406a8299f960e8409a92 in term 0.
I20260929 17:48:02.158346 8320 raft_consensus.cc:2454] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd392ede5942406a8299f960e8409a92 in term 0.
I20260929 17:48:02.158806 7989 leader_election.cc:302] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1414a591f2cf4a429331a47e060ec6be, cd392ede5942406a8299f960e8409a92; no voters:
I20260929 17:48:02.159065 8389 raft_consensus.cc:2781] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:02.159143 8389 raft_consensus.cc:482] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:02.159174 8389 raft_consensus.cc:3037] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:02.160338 8389 raft_consensus.cc:504] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.160655 8389 leader_election.cc:288] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [CANDIDATE]: Term 1 election: Requested vote from peers 129b7c9f5c0743dd9eadbf1256c5f2b6 (127.3.183.194:46711), 1414a591f2cf4a429331a47e060ec6be (127.3.183.195:32795)
I20260929 17:48:02.160988 8187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede6edaf923c49849070ce1a49f20fa4" candidate_uuid: "cd392ede5942406a8299f960e8409a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6"
I20260929 17:48:02.161056 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede6edaf923c49849070ce1a49f20fa4" candidate_uuid: "cd392ede5942406a8299f960e8409a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1414a591f2cf4a429331a47e060ec6be"
I20260929 17:48:02.161100 8187 raft_consensus.cc:3037] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:02.161175 8320 raft_consensus.cc:3037] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:02.162094 8187 raft_consensus.cc:2454] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd392ede5942406a8299f960e8409a92 in term 1.
I20260929 17:48:02.162212 8320 raft_consensus.cc:2454] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd392ede5942406a8299f960e8409a92 in term 1.
I20260929 17:48:02.162350 7989 leader_election.cc:302] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129b7c9f5c0743dd9eadbf1256c5f2b6, cd392ede5942406a8299f960e8409a92; no voters:
I20260929 17:48:02.162529 8389 raft_consensus.cc:2781] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:02.162729 8389 raft_consensus.cc:686] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [term 1 LEADER]: Becoming Leader. State: Replica: cd392ede5942406a8299f960e8409a92, State: Running, Role: LEADER
I20260929 17:48:02.163019 8389 consensus_queue.cc:237] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } }
I20260929 17:48:02.166180 7909 catalog_manager.cc:5495] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd392ede5942406a8299f960e8409a92 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "cd392ede5942406a8299f960e8409a92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd392ede5942406a8299f960e8409a92" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:02.186820 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.183379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.3.183.193:35155)
0929 17:48:02.183496 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.183500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.183513 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:02.183629 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.183632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.183684 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.183764 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:02.184050 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.184704 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.185524 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:02.185748 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:02.185817 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:02.186074 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:02.186484 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:02.186589 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.186640 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.186689 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 17:48:02.191511 8367 tablet.cc:2369] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:02.192643 8320 raft_consensus.cc:1261] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 1 FOLLOWER]: Refusing update from remote peer cd392ede5942406a8299f960e8409a92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:02.192647 8187 raft_consensus.cc:1261] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 1 FOLLOWER]: Refusing update from remote peer cd392ede5942406a8299f960e8409a92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:02.193289 8394 consensus_queue.cc:1035] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1414a591f2cf4a429331a47e060ec6be" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:02.193497 8389 consensus_queue.cc:1035] T ede6edaf923c49849070ce1a49f20fa4 P cd392ede5942406a8299f960e8409a92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129b7c9f5c0743dd9eadbf1256c5f2b6" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 46711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:02.200711 8399 mvcc.cc:205] Tried to move back new op lower bound from 7334723920653905920 to 7334723920542068736. Current Snapshot: MvccSnapshot[applied={T|T < 7334723920653905920}]
I20260929 17:48:02.201395 8398 mvcc.cc:205] Tried to move back new op lower bound from 7334723920653905920 to 7334723920542068736. Current Snapshot: MvccSnapshot[applied={T|T < 7334723920653905920}]
I20260929 17:48:02.208235 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:02.208419 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:02.210867 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60e773d7939a4e6ea9ca6a2a65f95ff4: Sending DeleteTablet for 3 replicas of tablet ede6edaf923c49849070ce1a49f20fa4
I20260929 17:48:02.211651 8034 tablet_service.cc:1514] Processing DeleteTablet for tablet ede6edaf923c49849070ce1a49f20fa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:02 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:51304
I20260929 17:48:02.211756 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet ede6edaf923c49849070ce1a49f20fa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:02 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:46036
I20260929 17:48:02.211952 8300 tablet_service.cc:1514] Processing DeleteTablet for tablet ede6edaf923c49849070ce1a49f20fa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:02 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42550
I20260929 17:48:02.212287 8407 tablet_replica.cc:330] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: stopping tablet replica
I20260929 17:48:02.212684 8407 raft_consensus.cc:2229] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:02.212816 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7972
I20260929 17:48:02.213505 8407 raft_consensus.cc:2258] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:02.214600 8408 tablet_replica.cc:330] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: stopping tablet replica
I20260929 17:48:02.214855 8408 raft_consensus.cc:2229] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:02.215142 8408 raft_consensus.cc:2258] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:02.216171 8407 ts_tablet_manager.cc:1905] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:02.218534 8407 ts_tablet_manager.cc:1918] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:02.218622 8407 log.cc:1196] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/ede6edaf923c49849070ce1a49f20fa4
I20260929 17:48:02.218910 8407 ts_tablet_manager.cc:1939] T ede6edaf923c49849070ce1a49f20fa4 P 129b7c9f5c0743dd9eadbf1256c5f2b6: Deleting consensus metadata
W20260929 17:48:02.219359 7897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:35155 (error 108)
I20260929 17:48:02.220115 8408 ts_tablet_manager.cc:1905] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:02.220232 7896 catalog_manager.cc:4841] TS 129b7c9f5c0743dd9eadbf1256c5f2b6 (127.3.183.194:46711): tablet ede6edaf923c49849070ce1a49f20fa4 (table test-table [id=e66c6a7aecd44fb2964478d880235e09]) successfully deleted
I20260929 17:48:02.220464 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.220115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:35155 (local address 127.0.0.1:51326)
0929 17:48:02.220303 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:02.220375 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:35155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 17:48:02.220619 7897 catalog_manager.cc:4584] TS cd392ede5942406a8299f960e8409a92 (127.3.183.193:35155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ede6edaf923c49849070ce1a49f20fa4: Network error: Client connection negotiation failed: client connection to 127.3.183.193:35155: connect: Connection refused (error 111)
I20260929 17:48:02.220854 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 8105
I20260929 17:48:02.222492 8408 ts_tablet_manager.cc:1918] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:02.222571 8408 log.cc:1196] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/ede6edaf923c49849070ce1a49f20fa4
I20260929 17:48:02.222851 8408 ts_tablet_manager.cc:1939] T ede6edaf923c49849070ce1a49f20fa4 P 1414a591f2cf4a429331a47e060ec6be: Deleting consensus metadata
I20260929 17:48:02.224638 7896 catalog_manager.cc:4841] TS 1414a591f2cf4a429331a47e060ec6be (127.3.183.195:32795): tablet ede6edaf923c49849070ce1a49f20fa4 (table test-table [id=e66c6a7aecd44fb2964478d880235e09]) successfully deleted
I20260929 17:48:02.228111 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 8238
I20260929 17:48:02.233103 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 7879
2026-09-29T17:48:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2263 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T17:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:02Z Disabled control of system clock
I20260929 17:48:02.261232 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:36325
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 17:48:02.370512 8413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:02.370779 8413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:02.370828 8413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:02.374780 8413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:02.374851 8413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:02.374873 8413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:02.374894 8413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:02.374913 8413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:02.379698 8413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:36325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:02.380775 8413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:02.382133 8413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:02.390161 8419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:02.390180 8418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:02.390174 8421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:02.390272 8413 server_base.cc:1034] running on GCE node
I20260929 17:48:02.390951 8413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:02.392015 8413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:02.393187 8413 hybrid_clock.cc:648] HybridClock initialized: now 1790704082393162 us; error 36 us; skew 500 ppm
I20260929 17:48:02.394953 8413 webserver.cc:460] Webserver started at http://127.3.183.254:35899/ using document root <none> and password file <none>
I20260929 17:48:02.395519 8413 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:02.395587 8413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:02.395802 8413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:02.397785 8413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "b6dc00f4d42a43e5829b924d9880a122"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "7a267063a8445a302651617303da07bc"
server_key_iv: "ac9695b5d78cf3641ab6f492f82346a5"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.398319 8413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "b6dc00f4d42a43e5829b924d9880a122"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "7a267063a8445a302651617303da07bc"
server_key_iv: "ac9695b5d78cf3641ab6f492f82346a5"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.401885 8413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:02.404357 8427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.405633 8413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:02.405748 8413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "b6dc00f4d42a43e5829b924d9880a122"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "7a267063a8445a302651617303da07bc"
server_key_iv: "ac9695b5d78cf3641ab6f492f82346a5"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.405848 8413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:02.410117 8413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:02.410676 8413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:02.410816 8413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:02.417282 8413 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:36325
I20260929 17:48:02.417300 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:36325 every 8 connection(s)
I20260929 17:48:02.418184 8413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:02.421389 8480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:02.426198 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 8413
I20260929 17:48:02.426324 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:02.426630 3807 external_mini_cluster.cc:1428] Setting key 500c5a49826e701a0c7b4b5929f02d96
I20260929 17:48:02.427223 8480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122: Bootstrap starting.
I20260929 17:48:02.429592 8480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:02.430377 8480 log.cc:824] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:02.432111 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.428112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.3.183.254:36325)
0929 17:48:02.428714 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.428728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.428764 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:48:02.429391 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.429409 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.429792 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.430067 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:02.430823 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:02.430828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:02.430844 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:02.430856 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:02.430921 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:02.430945 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:02.431048 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.431154 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.431393 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260929 17:48:02.432430 8480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122: No bootstrap required, opened a new log
I20260929 17:48:02.436043 8480 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dc00f4d42a43e5829b924d9880a122" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36325 } }
I20260929 17:48:02.436254 8480 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:02.436307 8480 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6dc00f4d42a43e5829b924d9880a122, State: Initialized, Role: FOLLOWER
I20260929 17:48:02.436689 8480 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dc00f4d42a43e5829b924d9880a122" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36325 } }
I20260929 17:48:02.436810 8480 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:02.436867 8480 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:02.436960 8480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:02.437798 8480 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dc00f4d42a43e5829b924d9880a122" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36325 } }
I20260929 17:48:02.438095 8480 leader_election.cc:302] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6dc00f4d42a43e5829b924d9880a122; no voters:
I20260929 17:48:02.438360 8480 leader_election.cc:288] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:02.438437 8485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:02.438643 8485 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [term 1 LEADER]: Becoming Leader. State: Replica: b6dc00f4d42a43e5829b924d9880a122, State: Running, Role: LEADER
I20260929 17:48:02.438972 8485 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dc00f4d42a43e5829b924d9880a122" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36325 } }
I20260929 17:48:02.439494 8480 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:02.440672 8487 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6dc00f4d42a43e5829b924d9880a122. Latest consensus state: current_term: 1 leader_uuid: "b6dc00f4d42a43e5829b924d9880a122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dc00f4d42a43e5829b924d9880a122" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36325 } } }
I20260929 17:48:02.440701 8486 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6dc00f4d42a43e5829b924d9880a122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dc00f4d42a43e5829b924d9880a122" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36325 } } }
I20260929 17:48:02.440791 8487 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:02.440791 8486 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:02.441108 8494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:02.444044 8494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:02.449460 8494 catalog_manager.cc:1347] Generated new cluster ID: d520e46e3fd74291af3270cd62bb6e94
I20260929 17:48:02.449515 8494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:02.465947 8494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:02.466290 8494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:02.469807 8494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122: Generated new TSK 0
I20260929 17:48:02.470427 8494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:48:02.478868 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36325
--builtin_ntp_servers=127.3.183.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 17:48:02.585225 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:02.585489 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:02.585544 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:02.589546 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:02.589604 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:02.589686 8504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:02.594420 8504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:02.595438 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:02.596817 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:02.606040 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:02.606034 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:02.606099 8504 server_base.cc:1034] running on GCE node
W20260929 17:48:02.606030 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:02.606559 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:02.607136 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:02.608296 8504 hybrid_clock.cc:648] HybridClock initialized: now 1790704082608273 us; error 33 us; skew 500 ppm
I20260929 17:48:02.610484 8504 webserver.cc:460] Webserver started at http://127.3.183.193:41037/ using document root <none> and password file <none>
I20260929 17:48:02.611177 8504 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:02.611244 8504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:02.611447 8504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:02.613551 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "d9234bf038284117b10ea219ad00dd84"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "cb3682ff4c9233b56e75f089a65705b1"
server_key_iv: "ada62082e0d46189361487e967bb5dfe"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.614096 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "d9234bf038284117b10ea219ad00dd84"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "cb3682ff4c9233b56e75f089a65705b1"
server_key_iv: "ada62082e0d46189361487e967bb5dfe"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.617712 8504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:02.619987 8518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.621148 8504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:02.621246 8504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "d9234bf038284117b10ea219ad00dd84"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "cb3682ff4c9233b56e75f089a65705b1"
server_key_iv: "ada62082e0d46189361487e967bb5dfe"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.621335 8504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:02.625643 8504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:02.626185 8504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:02.626327 8504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:02.626818 8504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:02.627681 8504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:02.627727 8504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.627774 8504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:02.627821 8504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.635918 8504 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:43345
I20260929 17:48:02.635952 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:43345 every 8 connection(s)
I20260929 17:48:02.636739 8504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:02.642457 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.638466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:46211 (local address 127.3.183.254:36325)
0929 17:48:02.638663 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.638668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.639653 (+ 985us) server_negotiation.cc:407] Connection header received
0929 17:48:02.641039 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.641042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.641078 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.641149 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:02.642151 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:02.642153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:02.642155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:02.642162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:02.642176 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:02.642183 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:02.642244 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.642320 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.642352 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:48:02.643273 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.638817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36325 (local address 127.3.183.193:46211)
0929 17:48:02.639486 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:02.639549 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:02.640853 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:02.641291 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:02.641297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:02.641383 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:02.641936 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:02.641937 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:02.641976 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:02.642043 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:02.642228 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:02.642232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:02.642235 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:48:02.642349 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 17:48:02.642541 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
I20260929 17:48:02.644520 8632 heartbeater.cc:347] Connected to a master server at 127.3.183.254:36325
I20260929 17:48:02.644691 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 8504
I20260929 17:48:02.644767 8632 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:02.644829 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:02.645141 3807 external_mini_cluster.cc:1428] Setting key e11ca8d566b8199f445fdaa38c7d2f9b
I20260929 17:48:02.645244 8632 heartbeater.cc:511] Master 127.3.183.254:36325 requested a full tablet report, sending...
I20260929 17:48:02.646714 8443 ts_manager.cc:194] Registered new tserver with Master: d9234bf038284117b10ea219ad00dd84 (127.3.183.193:43345)
I20260929 17:48:02.646992 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36325
--builtin_ntp_servers=127.3.183.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 17:48:02.755836 8635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:02.756105 8635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:02.756165 8635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:02.760164 8635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:02.760221 8635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:02.760295 8635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:02.765038 8635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:02.766089 8635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:02.767414 8635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:02.776922 8640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:02.777016 8635 server_base.cc:1034] running on GCE node
W20260929 17:48:02.776939 8643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:02.776918 8641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:02.777611 8635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:02.778282 8635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:02.779439 8635 hybrid_clock.cc:648] HybridClock initialized: now 1790704082779420 us; error 29 us; skew 500 ppm
I20260929 17:48:02.781224 8635 webserver.cc:460] Webserver started at http://127.3.183.194:42483/ using document root <none> and password file <none>
I20260929 17:48:02.781775 8635 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:02.781841 8635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:02.782030 8635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:02.784060 8635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "9ee15bc68f454d6aa770f30ddf3d2198"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "6b8ed5858fd1d2d851a972d751ebb2a6"
server_key_iv: "2ec61816d98a6b26276b29e24d342a7b"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.784585 8635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "9ee15bc68f454d6aa770f30ddf3d2198"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "6b8ed5858fd1d2d851a972d751ebb2a6"
server_key_iv: "2ec61816d98a6b26276b29e24d342a7b"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.788192 8635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 17:48:02.790400 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.791602 8635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:02.791728 8635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "9ee15bc68f454d6aa770f30ddf3d2198"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "6b8ed5858fd1d2d851a972d751ebb2a6"
server_key_iv: "2ec61816d98a6b26276b29e24d342a7b"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.791855 8635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:02.796274 8635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:02.796826 8635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:02.796970 8635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:02.797483 8635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:02.798328 8635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:02.798373 8635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.798422 8635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:02.798448 8635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.806327 8635 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:34171
I20260929 17:48:02.806375 8762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:34171 every 8 connection(s)
I20260929 17:48:02.807159 8635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:02.812762 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 8635
I20260929 17:48:02.812888 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:02.812974 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.808909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:46707 (local address 127.3.183.254:36325)
0929 17:48:02.809024 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.809028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.810127 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 17:48:02.811365 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.811368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.811412 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.811484 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:02.812484 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:02.812488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:02.812491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:02.812503 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:02.812526 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:02.812537 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:02.812633 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.812797 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.812840 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:48:02.813191 3807 external_mini_cluster.cc:1428] Setting key 41a4ffafa5fbf8f27b8358fd7bc1988c
I20260929 17:48:02.813881 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.809335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36325 (local address 127.3.183.194:46707)
0929 17:48:02.809976 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:02.810036 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:02.811175 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:02.811651 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:02.811657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:02.811742 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:02.812243 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:02.812245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:02.812282 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:02.812349 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:02.812651 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:02.812660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:02.812665 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:48:02.812813 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 17:48:02.813047 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260929 17:48:02.814844 8763 heartbeater.cc:347] Connected to a master server at 127.3.183.254:36325
I20260929 17:48:02.815070 8763 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:02.815156 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36325
--builtin_ntp_servers=127.3.183.212:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 17:48:02.815493 8763 heartbeater.cc:511] Master 127.3.183.254:36325 requested a full tablet report, sending...
I20260929 17:48:02.816505 8443 ts_manager.cc:194] Registered new tserver with Master: 9ee15bc68f454d6aa770f30ddf3d2198 (127.3.183.194:34171)
W20260929 17:48:02.924574 8766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:02.924847 8766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:02.924896 8766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:02.928960 8766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:02.929030 8766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:02.929088 8766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:02.934022 8766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:02.935098 8766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:02.936532 8766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:02.946280 8774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:02.946244 8771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:02.946295 8772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:02.946606 8766 server_base.cc:1034] running on GCE node
I20260929 17:48:02.947009 8766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:02.947625 8766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:02.948776 8766 hybrid_clock.cc:648] HybridClock initialized: now 1790704082948739 us; error 56 us; skew 500 ppm
I20260929 17:48:02.950572 8766 webserver.cc:460] Webserver started at http://127.3.183.195:34627/ using document root <none> and password file <none>
I20260929 17:48:02.951124 8766 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:02.951187 8766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:02.951368 8766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:02.953395 8766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "ad56af0c5a1743c494a463a9fe3cb390"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "e4595f654ae0b77f1c24fdc0162403d3"
server_key_iv: "ec95904cffdfc68f1488f7c58c5b6655"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.953902 8766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "ad56af0c5a1743c494a463a9fe3cb390"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "e4595f654ae0b77f1c24fdc0162403d3"
server_key_iv: "ec95904cffdfc68f1488f7c58c5b6655"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.957481 8766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 17:48:02.959676 8780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.961040 8766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:02.961159 8766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "ad56af0c5a1743c494a463a9fe3cb390"
format_stamp: "Formatted at 2026-09-29 17:48:02 on dist-test-slave-wbpt"
server_key: "e4595f654ae0b77f1c24fdc0162403d3"
server_key_iv: "ec95904cffdfc68f1488f7c58c5b6655"
server_key_version: "encryptionkey@0"
I20260929 17:48:02.961258 8766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:02.965754 8766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:02.966328 8766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:02.966482 8766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:02.966984 8766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:02.967911 8766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:02.967958 8766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.968007 8766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:02.968034 8766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:02.976423 8766 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:39687
I20260929 17:48:02.976455 8893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:39687 every 8 connection(s)
I20260929 17:48:02.977352 8766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:02.981007 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 8766
I20260929 17:48:02.981125 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:02.981356 3807 external_mini_cluster.cc:1428] Setting key ce73754f60ca9d55360ed7ea3c0e29f9
I20260929 17:48:02.983695 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.979218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:45799 (local address 127.3.183.254:36325)
0929 17:48:02.979387 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.979392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.980526 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 17:48:02.981919 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.981922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.981966 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.982048 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:02.983230 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:02.983233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:02.983236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:02.983248 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:02.983273 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:02.983282 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:02.983362 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.983491 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.983536 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 17:48:02.984529 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.979619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36325 (local address 127.3.183.195:45799)
0929 17:48:02.980360 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:02.980402 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:02.981710 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:02.982255 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:02.982264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:02.982376 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:02.982947 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:02.982949 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:02.982992 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:02.983061 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:02.983367 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:02.983373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:02.983378 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:48:02.983523 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 17:48:02.983765 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260929 17:48:02.985500 8894 heartbeater.cc:347] Connected to a master server at 127.3.183.254:36325
I20260929 17:48:02.985751 8894 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:02.986135 8894 heartbeater.cc:511] Master 127.3.183.254:36325 requested a full tablet report, sending...
I20260929 17:48:02.987075 8443 ts_manager.cc:194] Registered new tserver with Master: ad56af0c5a1743c494a463a9fe3cb390 (127.3.183.195:39687)
I20260929 17:48:02.995074 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:48:02.998464 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:02.996971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.3.183.254:36325)
0929 17:48:02.997100 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:02.997105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:02.997290 (+ 185us) server_negotiation.cc:407] Connection header received
0929 17:48:02.997405 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:02.997408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:02.997445 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:02.997508 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:02.998105 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:02.998108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:02.998111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:02.998123 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:02.998143 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:02.998151 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:02.998218 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:02.998300 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 17:48:02.998338 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:48:03.000947 8443 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:03.004058 8443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:03.015753 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.013190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.3.183.195:39687)
0929 17:48:03.014039 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.014050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.014072 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:48:03.014119 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.014124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.014294 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.014688 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.015205 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.015211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.015223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.015233 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.015274 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.015301 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.015409 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.015534 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.015610 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":150,"threads_started":1}
I20260929 17:48:03.015766 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.012435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:43345 (local address 127.0.0.1:39904)
0929 17:48:03.013269 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.013282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:03.013358 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:03.014333 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:03.014339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:03.014415 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:03.015219 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:03.015221 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:03.015229 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:03.015251 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:03.015589 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:03.015592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:03.015594 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 17:48:03.015654 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:48:03.015680 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":112,"threads_started":1}
I20260929 17:48:03.015740 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.012656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39687 (local address 127.0.0.1:59128)
0929 17:48:03.013580 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.013592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:03.013656 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:03.014446 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:03.014450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:03.014465 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:03.015006 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:03.015008 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:03.015028 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:03.015096 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:03.015384 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:03.015393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:03.015398 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:48:03.015514 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 17:48:03.015585 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":75,"threads_started":1}
I20260929 17:48:03.016062 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.012490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.3.183.194:34171)
0929 17:48:03.012904 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.012915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.013417 (+ 502us) server_negotiation.cc:407] Connection header received
0929 17:48:03.013481 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.013539 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.013779 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.013921 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.015266 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.015271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.015279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.015291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.015332 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.015365 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.015448 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.015839 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.015907 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":113,"threads_started":1}
I20260929 17:48:03.015743 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.012362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:34171 (local address 127.0.0.1:54274)
0929 17:48:03.013150 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.013199 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:03.013306 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:03.014619 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:03.014622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:03.014635 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:03.015108 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:03.015110 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:03.015121 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:03.015149 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:03.015497 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:03.015501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:03.015505 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:48:03.015575 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:48:03.015606 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":76,"threads_started":1}
I20260929 17:48:03.016546 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.013615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.3.183.193:43345)
0929 17:48:03.013972 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.013982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.014003 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:48:03.014045 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.014048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.014187 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.014312 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.015348 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.015354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.015365 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.015378 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.015420 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.015460 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.015553 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.016311 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.016378 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260929 17:48:03.019137 8697 tablet_service.cc:1467] Processing CreateTablet for tablet 6d44d3c7803a43979f545321efd1b497 (DEFAULT_TABLE table=test-table [id=0d908d554b9a4ee189f007d4d8c40789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:03.019160 8828 tablet_service.cc:1467] Processing CreateTablet for tablet 6d44d3c7803a43979f545321efd1b497 (DEFAULT_TABLE table=test-table [id=0d908d554b9a4ee189f007d4d8c40789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:03.019160 8566 tablet_service.cc:1467] Processing CreateTablet for tablet 6d44d3c7803a43979f545321efd1b497 (DEFAULT_TABLE table=test-table [id=0d908d554b9a4ee189f007d4d8c40789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:03.020208 8828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d44d3c7803a43979f545321efd1b497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:03.020208 8566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d44d3c7803a43979f545321efd1b497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:03.020210 8697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d44d3c7803a43979f545321efd1b497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:03.026206 8911 tablet_bootstrap.cc:492] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Bootstrap starting.
I20260929 17:48:03.027359 8913 tablet_bootstrap.cc:492] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: Bootstrap starting.
I20260929 17:48:03.027388 8912 tablet_bootstrap.cc:492] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Bootstrap starting.
I20260929 17:48:03.029343 8912 tablet_bootstrap.cc:654] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:03.029381 8911 tablet_bootstrap.cc:654] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:03.029381 8913 tablet_bootstrap.cc:654] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:03.030051 8912 log.cc:824] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:03.030159 8911 log.cc:824] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:03.030159 8913 log.cc:824] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:03.031888 8911 tablet_bootstrap.cc:492] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: No bootstrap required, opened a new log
I20260929 17:48:03.031898 8912 tablet_bootstrap.cc:492] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: No bootstrap required, opened a new log
I20260929 17:48:03.032078 8912 ts_tablet_manager.cc:1397] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:48:03.032080 8911 ts_tablet_manager.cc:1397] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 17:48:03.032956 8913 tablet_bootstrap.cc:492] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: No bootstrap required, opened a new log
I20260929 17:48:03.033115 8913 ts_tablet_manager.cc:1397] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 17:48:03.035616 8912 raft_consensus.cc:348] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.035878 8912 raft_consensus.cc:374] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:03.035954 8912 raft_consensus.cc:729] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad56af0c5a1743c494a463a9fe3cb390, State: Initialized, Role: FOLLOWER
I20260929 17:48:03.036374 8912 consensus_queue.cc:260] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.036345 8911 raft_consensus.cc:348] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.036618 8911 raft_consensus.cc:374] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:03.036711 8911 raft_consensus.cc:729] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee15bc68f454d6aa770f30ddf3d2198, State: Initialized, Role: FOLLOWER
I20260929 17:48:03.036983 8894 heartbeater.cc:503] Master 127.3.183.254:36325 was elected leader, sending a full tablet report...
I20260929 17:48:03.037197 8911 consensus_queue.cc:260] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.037344 8912 ts_tablet_manager.cc:1428] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:48:03.037880 8763 heartbeater.cc:503] Master 127.3.183.254:36325 was elected leader, sending a full tablet report...
I20260929 17:48:03.038131 8911 ts_tablet_manager.cc:1428] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:48:03.038331 8913 raft_consensus.cc:348] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.038537 8913 raft_consensus.cc:374] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:03.038615 8913 raft_consensus.cc:729] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9234bf038284117b10ea219ad00dd84, State: Initialized, Role: FOLLOWER
I20260929 17:48:03.038985 8913 consensus_queue.cc:260] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.039561 8632 heartbeater.cc:503] Master 127.3.183.254:36325 was elected leader, sending a full tablet report...
I20260929 17:48:03.039947 8913 ts_tablet_manager.cc:1428] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 17:48:03.058044 8764 tablet.cc:2369] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:48:03.137861 8633 tablet.cc:2369] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:03.225648 8918 raft_consensus.cc:482] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:03.225850 8918 raft_consensus.cc:504] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.226965 8918 leader_election.cc:288] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9234bf038284117b10ea219ad00dd84 (127.3.183.193:43345), ad56af0c5a1743c494a463a9fe3cb390 (127.3.183.195:39687)
W20260929 17:48:03.228286 8895 tablet.cc:2369] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:03.228796 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.227212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:43345 (local address 127.3.183.194:35389)
0929 17:48:03.227386 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.227400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:03.227503 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:03.227730 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:03.227733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:03.227748 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:03.228259 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:03.228261 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:03.228271 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:03.228296 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:03.228545 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:03.228548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:03.228552 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:48:03.228632 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:48:03.228677 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 17:48:03.228915 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.227267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:35389 (local address 127.3.183.193:43345)
0929 17:48:03.227387 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.227391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.227470 (+ 79us) server_negotiation.cc:407] Connection header received
0929 17:48:03.227596 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.227598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.227631 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.227696 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.228399 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.228403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.228406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.228418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.228439 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.228450 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.228540 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.228745 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.228788 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:48:03.229029 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.227327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:52711 (local address 127.3.183.195:39687)
0929 17:48:03.227451 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.227456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.227752 (+ 296us) server_negotiation.cc:407] Connection header received
0929 17:48:03.227858 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.227861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.227895 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.227995 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.228624 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.228627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.228629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.228642 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.228663 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.228673 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.228745 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.228878 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.228914 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:48:03.229321 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.227265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39687 (local address 127.3.183.194:52711)
0929 17:48:03.227630 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.227642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:03.227755 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:03.228040 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:03.228043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:03.228058 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:03.228425 (+ 367us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:03.228427 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:03.228436 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:03.228453 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:03.228782 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:03.228785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:03.228788 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:48:03.229159 (+ 371us) client_negotiation.cc:240] Negotiation successful
0929 17:48:03.229194 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":148,"threads_started":1}
I20260929 17:48:03.229592 8586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d44d3c7803a43979f545321efd1b497" candidate_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9234bf038284117b10ea219ad00dd84" is_pre_election: true
I20260929 17:48:03.229894 8586 raft_consensus.cc:2454] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ee15bc68f454d6aa770f30ddf3d2198 in term 0.
I20260929 17:48:03.229936 8848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d44d3c7803a43979f545321efd1b497" candidate_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56af0c5a1743c494a463a9fe3cb390" is_pre_election: true
I20260929 17:48:03.230185 8848 raft_consensus.cc:2454] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ee15bc68f454d6aa770f30ddf3d2198 in term 0.
I20260929 17:48:03.230368 8652 leader_election.cc:302] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee15bc68f454d6aa770f30ddf3d2198, d9234bf038284117b10ea219ad00dd84; no voters:
I20260929 17:48:03.230635 8918 raft_consensus.cc:2781] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:03.230717 8918 raft_consensus.cc:482] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:03.230749 8918 raft_consensus.cc:3037] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:03.231753 8918 raft_consensus.cc:504] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.232123 8918 leader_election.cc:288] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [CANDIDATE]: Term 1 election: Requested vote from peers d9234bf038284117b10ea219ad00dd84 (127.3.183.193:43345), ad56af0c5a1743c494a463a9fe3cb390 (127.3.183.195:39687)
I20260929 17:48:03.232424 8848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d44d3c7803a43979f545321efd1b497" candidate_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56af0c5a1743c494a463a9fe3cb390"
I20260929 17:48:03.232436 8586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d44d3c7803a43979f545321efd1b497" candidate_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9234bf038284117b10ea219ad00dd84"
I20260929 17:48:03.232545 8848 raft_consensus.cc:3037] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:03.232546 8586 raft_consensus.cc:3037] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:03.233737 8848 raft_consensus.cc:2454] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ee15bc68f454d6aa770f30ddf3d2198 in term 1.
I20260929 17:48:03.233737 8586 raft_consensus.cc:2454] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ee15bc68f454d6aa770f30ddf3d2198 in term 1.
I20260929 17:48:03.234030 8650 leader_election.cc:302] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9ee15bc68f454d6aa770f30ddf3d2198, ad56af0c5a1743c494a463a9fe3cb390, d9234bf038284117b10ea219ad00dd84; no voters:
I20260929 17:48:03.234196 8918 raft_consensus.cc:2781] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:03.234396 8918 raft_consensus.cc:686] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 1 LEADER]: Becoming Leader. State: Replica: 9ee15bc68f454d6aa770f30ddf3d2198, State: Running, Role: LEADER
I20260929 17:48:03.234840 8918 consensus_queue.cc:237] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } }
I20260929 17:48:03.238029 8442 catalog_manager.cc:5495] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ee15bc68f454d6aa770f30ddf3d2198 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee15bc68f454d6aa770f30ddf3d2198" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 34171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:03.250407 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.248964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.3.183.194:34171)
0929 17:48:03.249098 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.249102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.249117 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:48:03.249233 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.249235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.249296 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.249371 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.250050 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.250054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.250057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.250069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.250089 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.250098 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.250174 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.250239 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.250279 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:48:03.256120 8586 raft_consensus.cc:1261] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 1 FOLLOWER]: Refusing update from remote peer 9ee15bc68f454d6aa770f30ddf3d2198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:03.256119 8848 raft_consensus.cc:1261] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 1 FOLLOWER]: Refusing update from remote peer 9ee15bc68f454d6aa770f30ddf3d2198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:03.256775 8921 consensus_queue.cc:1035] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9234bf038284117b10ea219ad00dd84" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 43345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:03.256958 8918 consensus_queue.cc:1035] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad56af0c5a1743c494a463a9fe3cb390" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:03.271478 8444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:03.271654 8444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:03.273940 8444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6dc00f4d42a43e5829b924d9880a122: Sending DeleteTablet for 3 replicas of tablet 6d44d3c7803a43979f545321efd1b497
I20260929 17:48:03.274638 8697 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d44d3c7803a43979f545321efd1b497 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:03 UTC) from {username='slave'} at 127.0.0.1:54274
I20260929 17:48:03.274632 8566 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d44d3c7803a43979f545321efd1b497 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:03 UTC) from {username='slave'} at 127.0.0.1:39904
I20260929 17:48:03.274722 8828 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d44d3c7803a43979f545321efd1b497 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:03 UTC) from {username='slave'} at 127.0.0.1:59128
I20260929 17:48:03.275215 8933 tablet_replica.cc:330] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84: stopping tablet replica
I20260929 17:48:03.275509 8933 raft_consensus.cc:2229] T 6d44d3c7803a43979f545321efd1b497 P d9234bf038284117b10ea219ad00dd84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:03.275695 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 8504
I20260929 17:48:03.276700 8935 tablet_replica.cc:330] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: stopping tablet replica
I20260929 17:48:03.276995 8935 raft_consensus.cc:2229] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:48:03.277360 8935 raft_consensus.cc:2258] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:03.278587 8935 ts_tablet_manager.cc:1905] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:03.279995 8934 tablet_replica.cc:330] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: stopping tablet replica
I20260929 17:48:03.280310 8934 raft_consensus.cc:2229] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:03.280651 8934 raft_consensus.cc:2258] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:03.281437 8935 ts_tablet_manager.cc:1918] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:03.281570 8935 log.cc:1196] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/6d44d3c7803a43979f545321efd1b497
I20260929 17:48:03.281950 8935 ts_tablet_manager.cc:1939] T 6d44d3c7803a43979f545321efd1b497 P 9ee15bc68f454d6aa770f30ddf3d2198: Deleting consensus metadata
I20260929 17:48:03.282120 8934 ts_tablet_manager.cc:1905] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:48:03.282438 8430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:43345 (error 108)
I20260929 17:48:03.283149 8431 catalog_manager.cc:4841] TS 9ee15bc68f454d6aa770f30ddf3d2198 (127.3.183.194:34171): tablet 6d44d3c7803a43979f545321efd1b497 (table test-table [id=0d908d554b9a4ee189f007d4d8c40789]) successfully deleted
I20260929 17:48:03.283653 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.283411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:43345 (local address 127.0.0.1:39918)
0929 17:48:03.283513 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.283578 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 17:48:03.283821 8430 catalog_manager.cc:4584] TS d9234bf038284117b10ea219ad00dd84 (127.3.183.193:43345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d44d3c7803a43979f545321efd1b497: Network error: Client connection negotiation failed: client connection to 127.3.183.193:43345: connect: Connection refused (error 111)
I20260929 17:48:03.284129 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 8635
I20260929 17:48:03.285073 8934 ts_tablet_manager.cc:1918] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:03.285140 8934 log.cc:1196] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/6d44d3c7803a43979f545321efd1b497
I20260929 17:48:03.285327 8934 ts_tablet_manager.cc:1939] T 6d44d3c7803a43979f545321efd1b497 P ad56af0c5a1743c494a463a9fe3cb390: Deleting consensus metadata
I20260929 17:48:03.286504 8428 catalog_manager.cc:4841] TS ad56af0c5a1743c494a463a9fe3cb390 (127.3.183.195:39687): tablet 6d44d3c7803a43979f545321efd1b497 (table test-table [id=0d908d554b9a4ee189f007d4d8c40789]) successfully deleted
I20260929 17:48:03.289229 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 8766
I20260929 17:48:03.293612 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 8413
2026-09-29T17:48:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1059 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 17:48:03.313776 3807 mini_oidc.cc:152] Starting JWKS server
I20260929 17:48:03.315399 3807 webserver.cc:460] Webserver started at https://127.0.0.1:40481/ using document root <none> and password file <none>
I20260929 17:48:03.315582 3807 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:48:03.315882 3807 webserver.cc:460] Webserver started at http://127.0.0.1:42867/ using document root <none> and password file <none>
2026-09-29T17:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:03Z Disabled control of system clock
I20260929 17:48:03.324739 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:43515
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:43515
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40481/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/testchainca.cert with env {}
W20260929 17:48:03.430589 8942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:03.430850 8942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:03.430888 8942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:03.434902 8942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:03.434966 8942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:03.434981 8942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:03.434996 8942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:03.435012 8942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:48:03.435030 8942 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:48:03.435048 8942 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40481/jwks.json
I20260929 17:48:03.440323 8942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:43515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:43515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40481/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:03.441262 8942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:03.442258 8942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:03.450456 8947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:03.450592 8942 server_base.cc:1034] running on GCE node
W20260929 17:48:03.450443 8950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:03.450431 8948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:03.451363 8942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:03.452454 8942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:03.453608 8942 hybrid_clock.cc:648] HybridClock initialized: now 1790704083453591 us; error 31 us; skew 500 ppm
I20260929 17:48:03.455304 8942 webserver.cc:460] Webserver started at http://127.3.183.254:42487/ using document root <none> and password file <none>
I20260929 17:48:03.455868 8942 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:03.455938 8942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:03.456128 8942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:03.458052 8942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "0b637ccf3ddd4d29998dc9b6a3f85047"
format_stamp: "Formatted at 2026-09-29 17:48:03 on dist-test-slave-wbpt"
server_key: "992862abb28d7d45d814ea909a659068"
server_key_iv: "a83c2201548d59881361ced255b1e5f0"
server_key_version: "encryptionkey@0"
I20260929 17:48:03.458571 8942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "0b637ccf3ddd4d29998dc9b6a3f85047"
format_stamp: "Formatted at 2026-09-29 17:48:03 on dist-test-slave-wbpt"
server_key: "992862abb28d7d45d814ea909a659068"
server_key_iv: "a83c2201548d59881361ced255b1e5f0"
server_key_version: "encryptionkey@0"
I20260929 17:48:03.462023 8942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:03.464246 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:03.465406 8942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:03.465517 8942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "0b637ccf3ddd4d29998dc9b6a3f85047"
format_stamp: "Formatted at 2026-09-29 17:48:03 on dist-test-slave-wbpt"
server_key: "992862abb28d7d45d814ea909a659068"
server_key_iv: "a83c2201548d59881361ced255b1e5f0"
server_key_version: "encryptionkey@0"
I20260929 17:48:03.465603 8942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:03.500948 8942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:03.501669 8942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:03.501847 8942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:03.508558 8942 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:43515
I20260929 17:48:03.508594 9008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:43515 every 8 connection(s)
I20260929 17:48:03.509447 8942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:03.509801 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 8942
I20260929 17:48:03.509908 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:03.510191 3807 external_mini_cluster.cc:1428] Setting key b302488198a7576ff23ec0bab04fba42
I20260929 17:48:03.513041 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:03.519078 9009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047: Bootstrap starting.
I20260929 17:48:03.519326 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.511458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.3.183.254:43515)
0929 17:48:03.512096 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.512112 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.512152 (+ 40us) server_negotiation.cc:407] Connection header received
0929 17:48:03.512906 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.512924 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.513347 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.513640 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.513975 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.514925 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.516136 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.516610 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.516729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:03.517555 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.517560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.517572 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.517935 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.517988 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.518010 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.518080 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.518278 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.518592 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260929 17:48:03.521441 9009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:03.522042 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:48:03.522128 9009 log.cc:824] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:03.523847 9009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047: No bootstrap required, opened a new log
I20260929 17:48:03.527063 9009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43515 } }
I20260929 17:48:03.527279 9009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:03.527341 9009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b637ccf3ddd4d29998dc9b6a3f85047, State: Initialized, Role: FOLLOWER
I20260929 17:48:03.527724 9009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43515 } }
I20260929 17:48:03.527878 9009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:03.527946 9009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:03.528047 9009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:03.528908 9009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43515 } }
I20260929 17:48:03.529191 9009 leader_election.cc:302] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b637ccf3ddd4d29998dc9b6a3f85047; no voters:
I20260929 17:48:03.529422 9009 leader_election.cc:288] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:03.529597 9014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:03.529826 9014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [term 1 LEADER]: Becoming Leader. State: Replica: 0b637ccf3ddd4d29998dc9b6a3f85047, State: Running, Role: LEADER
I20260929 17:48:03.530123 9014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43515 } }
I20260929 17:48:03.530532 9009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:03.531713 9016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b637ccf3ddd4d29998dc9b6a3f85047. Latest consensus state: current_term: 1 leader_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43515 } } }
I20260929 17:48:03.531772 9015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b637ccf3ddd4d29998dc9b6a3f85047" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43515 } } }
I20260929 17:48:03.531857 9016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:03.531857 9015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:03.532253 9023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:03.535292 9023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:03.540863 9023 catalog_manager.cc:1347] Generated new cluster ID: 6a430a69196345e99c54441a607de66d
I20260929 17:48:03.540934 9023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:03.563298 9023 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:03.564388 9023 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:03.572144 9023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b637ccf3ddd4d29998dc9b6a3f85047: Generated new TSK 0
I20260929 17:48:03.572867 9023 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 17:48:03.610363 9039 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 17:48:03.610563 9039 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 17:48:03.611074 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.586334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.3.183.254:43515)
0929 17:48:03.586453 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.586458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.586679 (+ 221us) server_negotiation.cc:407] Connection header received
0929 17:48:03.586891 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.586894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.586943 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.587026 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:48:03.587281 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.587840 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.588680 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.588887 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.588970 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:03.589325 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:48:03.610653 (+ 21328us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 17:48:03.610934 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 17:48:03.611150 9012 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 17:48:03.612485 9038 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:03.586268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:43515 (local address 127.0.0.1:56298)
0929 17:48:03.586590 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.586601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:03.586732 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:03.587047 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:03.587051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:03.587058 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:48:03.587179 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:03.587186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.587938 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.587942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:03.588568 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:03.588575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.588986 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.588990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:03.589084 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:03.589145 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:48:03.610895 (+ 21750us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 17:48:03.611157 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.183.254:43515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":111,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.183.254:43515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40481/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 17:48:03.613658 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 8942
2026-09-29T17:48:03Z chronyd exiting
I20260929 17:48:03.651101 3807 test_util.cc:175] -----------------------------------------------
I20260929 17:48:03.651211 3807 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704058755242-3807-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 17:48:03.652441 3807 mini_oidc.cc:152] Starting JWKS server
I20260929 17:48:03.653302 3807 webserver.cc:460] Webserver started at https://127.0.0.1:36397/ using document root <none> and password file <none>
I20260929 17:48:03.653385 3807 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:48:03.653606 3807 webserver.cc:460] Webserver started at http://127.0.0.1:46541/ using document root <none> and password file <none>
2026-09-29T17:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:03Z Disabled control of system clock
I20260929 17:48:03.662494 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:37053
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:37053
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/testchainca.cert with env {}
W20260929 17:48:03.771504 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:03.771806 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:03.771864 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:03.775915 9045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:03.775983 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:03.775997 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:03.776013 9045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:03.776027 9045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:48:03.776043 9045 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:48:03.776059 9045 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36397/jwks.json
I20260929 17:48:03.781626 9045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:37053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:37053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:03.782712 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:03.783708 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:03.792218 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:03.792273 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:03.792265 9045 server_base.cc:1034] running on GCE node
W20260929 17:48:03.792220 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:03.793100 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:03.794247 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:03.795410 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790704083795395 us; error 29 us; skew 500 ppm
I20260929 17:48:03.797288 9045 webserver.cc:460] Webserver started at http://127.3.183.254:33027/ using document root <none> and password file <none>
I20260929 17:48:03.797848 9045 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:03.797915 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:03.798117 9045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:03.800220 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "fc0b89eaf67041a4b9a0631af1109c66"
format_stamp: "Formatted at 2026-09-29 17:48:03 on dist-test-slave-wbpt"
server_key: "9ab3dcb4720fe2f8c9c3ed1dec6efa90"
server_key_iv: "37e58b02ef235a8eaec36cf1310b1506"
server_key_version: "encryptionkey@0"
I20260929 17:48:03.800757 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "fc0b89eaf67041a4b9a0631af1109c66"
format_stamp: "Formatted at 2026-09-29 17:48:03 on dist-test-slave-wbpt"
server_key: "9ab3dcb4720fe2f8c9c3ed1dec6efa90"
server_key_iv: "37e58b02ef235a8eaec36cf1310b1506"
server_key_version: "encryptionkey@0"
I20260929 17:48:03.804579 9045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:48:03.806883 9059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:03.808110 9045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:03.808216 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "fc0b89eaf67041a4b9a0631af1109c66"
format_stamp: "Formatted at 2026-09-29 17:48:03 on dist-test-slave-wbpt"
server_key: "9ab3dcb4720fe2f8c9c3ed1dec6efa90"
server_key_iv: "37e58b02ef235a8eaec36cf1310b1506"
server_key_version: "encryptionkey@0"
I20260929 17:48:03.808314 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:03.842717 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:03.843469 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:03.843645 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:03.850430 9045 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:37053
I20260929 17:48:03.850436 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:37053 every 8 connection(s)
I20260929 17:48:03.851431 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:03.854471 9112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:03.858269 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9045
I20260929 17:48:03.858397 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:03.858687 3807 external_mini_cluster.cc:1428] Setting key b099f69e5825c8d2e3e9c737c644d0ba
I20260929 17:48:03.860843 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66: Bootstrap starting.
I20260929 17:48:03.863071 9112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:03.863797 9112 log.cc:824] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:03.865543 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66: No bootstrap required, opened a new log
I20260929 17:48:03.867883 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.860143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.3.183.254:37053)
0929 17:48:03.860847 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.860867 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.860902 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:48:03.861593 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.861611 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.862021 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.862330 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:03.862760 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.863804 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.864658 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.865118 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.865266 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:03.866032 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:03.866037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:03.866050 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:03.866435 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:03.866495 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:03.866515 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:03.866598 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:03.866740 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 17:48:03.866977 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":126,"threads_started":1}
I20260929 17:48:03.868605 9112 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b89eaf67041a4b9a0631af1109c66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37053 } }
I20260929 17:48:03.868871 9112 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:03.868914 9112 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0b89eaf67041a4b9a0631af1109c66, State: Initialized, Role: FOLLOWER
I20260929 17:48:03.869272 9112 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b89eaf67041a4b9a0631af1109c66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37053 } }
I20260929 17:48:03.869375 9112 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:03.869412 9112 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:03.869482 9112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:03.870767 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:48:03.940328 9112 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b89eaf67041a4b9a0631af1109c66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37053 } }
I20260929 17:48:03.940873 9112 leader_election.cc:302] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0b89eaf67041a4b9a0631af1109c66; no voters:
I20260929 17:48:03.941272 9112 leader_election.cc:288] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:03.941416 9117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:03.941660 9117 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [term 1 LEADER]: Becoming Leader. State: Replica: fc0b89eaf67041a4b9a0631af1109c66, State: Running, Role: LEADER
I20260929 17:48:03.942024 9117 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b89eaf67041a4b9a0631af1109c66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37053 } }
I20260929 17:48:03.942698 9112 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:03.943642 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0b89eaf67041a4b9a0631af1109c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b89eaf67041a4b9a0631af1109c66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37053 } } }
I20260929 17:48:03.943799 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:03.944170 9126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:03.944180 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0b89eaf67041a4b9a0631af1109c66. Latest consensus state: current_term: 1 leader_uuid: "fc0b89eaf67041a4b9a0631af1109c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b89eaf67041a4b9a0631af1109c66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37053 } } }
I20260929 17:48:03.944588 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:03.947301 9126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:03.952821 9126 catalog_manager.cc:1347] Generated new cluster ID: 1cb6ca32eb884bec882351f9615c935d
I20260929 17:48:03.952889 9126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:03.968883 9126 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:03.970165 9126 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:03.984498 9126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc0b89eaf67041a4b9a0631af1109c66: Generated new TSK 0
I20260929 17:48:03.985415 9126 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 17:48:04.013262 9142 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 17:48:04.013339 9142 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 17:48:04.014096 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:03.997847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.3.183.254:37053)
0929 17:48:03.997973 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:03.997978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:03.998111 (+ 133us) server_negotiation.cc:407] Connection header received
0929 17:48:03.998259 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:03.998263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:03.998319 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:03.998448 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:48:03.998791 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.999607 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.000523 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.000874 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.000973 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.001302 (+ 329us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:48:04.013662 (+ 12360us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 17:48:04.013947 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46524: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 17:48:04.014165 9115 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46524: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 17:48:04.014189 9141 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:03.997737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:37053 (local address 127.0.0.1:46524)
0929 17:48:03.998019 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:03.998031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:03.998135 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:03.998502 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:03.998505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:03.998514 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:48:03.998675 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:03.998681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:03.999736 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:03.999739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:04.000413 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:04.000422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.001007 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.001010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:04.001099 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.001124 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:48:04.013865 (+ 12741us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 17:48:04.014014 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.183.254:37053: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260929 17:48:04.015128 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9045
2026-09-29T17:48:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (401 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 17:48:04.053948 3807 mini_oidc.cc:152] Starting JWKS server
I20260929 17:48:04.054834 3807 webserver.cc:460] Webserver started at https://127.0.0.1:37043/ using document root <none> and password file <none>
I20260929 17:48:04.054924 3807 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:48:04.055173 3807 webserver.cc:460] Webserver started at http://127.0.0.1:37071/ using document root <none> and password file <none>
2026-09-29T17:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:04Z Disabled control of system clock
I20260929 17:48:04.063694 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:40449
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:42915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:40449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37043/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 17:48:04.171319 9148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:04.171598 9148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:04.171661 9148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:04.175642 9148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:04.175704 9148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:04.175724 9148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:04.175745 9148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:04.175765 9148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:48:04.175818 9148 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:48:04.175841 9148 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37043/jwks.json
I20260929 17:48:04.180548 9148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:40449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:40449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37043/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:04.181584 9148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:04.182928 9148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:04.190939 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:04.190948 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:04.190944 9156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:04.190986 9148 server_base.cc:1034] running on GCE node
I20260929 17:48:04.191726 9148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:04.192812 9148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:04.193969 9148 hybrid_clock.cc:648] HybridClock initialized: now 1790704084193948 us; error 36 us; skew 500 ppm
I20260929 17:48:04.200878 9148 webserver.cc:460] Webserver started at http://127.3.183.254:41031/ using document root <none> and password file <none>
I20260929 17:48:04.201519 9148 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:04.201588 9148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:04.201787 9148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:04.204149 9148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "58b78f91c82845818bedf79e5da42e5b"
format_stamp: "Formatted at 2026-09-29 17:48:04 on dist-test-slave-wbpt"
server_key: "9169ff45fe899be32a80f94f5b546420"
server_key_iv: "14055a76e3ae8e1924105cca3632d614"
server_key_version: "encryptionkey@0"
I20260929 17:48:04.204731 9148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "58b78f91c82845818bedf79e5da42e5b"
format_stamp: "Formatted at 2026-09-29 17:48:04 on dist-test-slave-wbpt"
server_key: "9169ff45fe899be32a80f94f5b546420"
server_key_iv: "14055a76e3ae8e1924105cca3632d614"
server_key_version: "encryptionkey@0"
I20260929 17:48:04.208587 9148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 17:48:04.211237 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:04.212567 9148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:48:04.212715 9148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "58b78f91c82845818bedf79e5da42e5b"
format_stamp: "Formatted at 2026-09-29 17:48:04 on dist-test-slave-wbpt"
server_key: "9169ff45fe899be32a80f94f5b546420"
server_key_iv: "14055a76e3ae8e1924105cca3632d614"
server_key_version: "encryptionkey@0"
I20260929 17:48:04.212819 9148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:04.223740 9148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:04.224414 9148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:04.224576 9148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:04.230947 9148 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:40449
I20260929 17:48:04.230964 9214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:40449 every 8 connection(s)
I20260929 17:48:04.231901 9148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:04.235024 9215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:04.239483 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9148
I20260929 17:48:04.239581 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:04.239864 3807 external_mini_cluster.cc:1428] Setting key bb43d56fd4a3b1c900aad365717e4e0a
I20260929 17:48:04.240712 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b: Bootstrap starting.
I20260929 17:48:04.243427 9215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:04.244376 9215 log.cc:824] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:04.246287 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b: No bootstrap required, opened a new log
I20260929 17:48:04.249776 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:04.241257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.3.183.254:40449)
0929 17:48:04.241898 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:04.241912 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:04.241947 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:48:04.242574 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:04.242591 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:04.243001 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:04.243319 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:04.243818 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.244849 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.246021 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.246537 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.246742 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.247693 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:04.247700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:04.247716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:04.248236 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:04.248307 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:04.248333 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:04.248430 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:04.248524 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 17:48:04.248799 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260929 17:48:04.250754 9215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b78f91c82845818bedf79e5da42e5b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40449 } }
I20260929 17:48:04.251016 9215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:04.251057 9215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b78f91c82845818bedf79e5da42e5b, State: Initialized, Role: FOLLOWER
I20260929 17:48:04.251480 9215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b78f91c82845818bedf79e5da42e5b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40449 } }
I20260929 17:48:04.251605 9215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:04.251683 9215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:04.251770 9215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:04.252741 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:48:04.344098 9215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b78f91c82845818bedf79e5da42e5b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40449 } }
I20260929 17:48:04.344565 9215 leader_election.cc:302] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58b78f91c82845818bedf79e5da42e5b; no voters:
I20260929 17:48:04.344940 9215 leader_election.cc:288] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:04.345114 9220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:04.345391 9220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [term 1 LEADER]: Becoming Leader. State: Replica: 58b78f91c82845818bedf79e5da42e5b, State: Running, Role: LEADER
I20260929 17:48:04.346421 9220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b78f91c82845818bedf79e5da42e5b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40449 } }
I20260929 17:48:04.346537 9215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:04.348731 9221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58b78f91c82845818bedf79e5da42e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b78f91c82845818bedf79e5da42e5b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40449 } } }
I20260929 17:48:04.349195 9221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:04.348796 9222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b78f91c82845818bedf79e5da42e5b. Latest consensus state: current_term: 1 leader_uuid: "58b78f91c82845818bedf79e5da42e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b78f91c82845818bedf79e5da42e5b" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40449 } } }
I20260929 17:48:04.349442 9222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b [sys.catalog]: This master's current role is: LEADER
W20260929 17:48:04.351418 9234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:48:04.351493 9234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:48:04.351568 9235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:04.352262 9235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:04.357290 9235 catalog_manager.cc:1347] Generated new cluster ID: 3213cbfbd6eb45c29008b66b003ded34
I20260929 17:48:04.357422 9235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:04.382073 9235 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:04.383029 9235 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:04.390898 9235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58b78f91c82845818bedf79e5da42e5b: Generated new TSK 0
I20260929 17:48:04.391625 9235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:48:04.421928 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:04.416182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.3.183.254:40449)
0929 17:48:04.416380 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:04.416385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:04.416615 (+ 230us) server_negotiation.cc:407] Connection header received
0929 17:48:04.416870 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:04.416872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:04.416926 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:04.417031 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:48:04.417303 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.417920 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.419097 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.419385 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.419490 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.419967 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:48:04.421505 (+ 1538us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 17:48:04.421805 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 17:48:04.421995 9218 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 17:48:04.424593 9244 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:04.415992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:40449 (local address 127.0.0.1:34410)
0929 17:48:04.416499 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:04.416516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:04.416714 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:04.417055 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:04.417058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:04.417066 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:48:04.417192 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:04.417198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.418090 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.418094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:04.418974 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:04.418984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.419517 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.419522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:04.419653 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.419685 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:48:04.421704 (+ 2019us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 17:48:04.424448 (+ 2744us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.183.254:40449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":2568,"thread_start_us":112,"threads_started":1}
I20260929 17:48:04.425179 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9148
E20260929 17:48:04.435822 9245 webserver.cc:533] Webserver: sslize failed:
E20260929 17:48:04.435911 9245 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T17:48:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (410 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T17:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:04Z Disabled control of system clock
I20260929 17:48:04.474345 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:42499
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:42499
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/testchainca.cert with env {}
W20260929 17:48:04.584053 9249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:04.584328 9249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:04.584373 9249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:04.588505 9249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:04.588572 9249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:04.588585 9249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:04.588601 9249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:04.588626 9249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:48:04.588655 9249 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:48:04.588672 9249 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 17:48:04.594086 9249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:42499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:04.595129 9249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:04.596133 9249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:04.604489 9254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:04.604555 9249 server_base.cc:1034] running on GCE node
W20260929 17:48:04.604583 9257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:04.604583 9255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:04.605312 9249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:04.606392 9249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:04.607548 9249 hybrid_clock.cc:648] HybridClock initialized: now 1790704084607532 us; error 31 us; skew 500 ppm
I20260929 17:48:04.609390 9249 webserver.cc:460] Webserver started at http://127.3.183.254:40159/ using document root <none> and password file <none>
I20260929 17:48:04.609926 9249 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:04.609985 9249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:04.610172 9249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:04.612211 9249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "c41a8724f54e45d09a132f0b567927b3"
format_stamp: "Formatted at 2026-09-29 17:48:04 on dist-test-slave-wbpt"
server_key: "036e68eb68d90a1871f61689463ff0c2"
server_key_iv: "7166e53017d965fef748c2fe44337b22"
server_key_version: "encryptionkey@0"
I20260929 17:48:04.612747 9249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "c41a8724f54e45d09a132f0b567927b3"
format_stamp: "Formatted at 2026-09-29 17:48:04 on dist-test-slave-wbpt"
server_key: "036e68eb68d90a1871f61689463ff0c2"
server_key_iv: "7166e53017d965fef748c2fe44337b22"
server_key_version: "encryptionkey@0"
I20260929 17:48:04.616282 9249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:04.618641 9263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:04.619876 9249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:04.619997 9249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "c41a8724f54e45d09a132f0b567927b3"
format_stamp: "Formatted at 2026-09-29 17:48:04 on dist-test-slave-wbpt"
server_key: "036e68eb68d90a1871f61689463ff0c2"
server_key_iv: "7166e53017d965fef748c2fe44337b22"
server_key_version: "encryptionkey@0"
I20260929 17:48:04.620097 9249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:04.643445 9249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:04.644199 9249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:04.644376 9249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:04.651254 9249 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:42499
I20260929 17:48:04.651281 9315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:42499 every 8 connection(s)
I20260929 17:48:04.652283 9249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:04.655508 9316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:04.660136 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9249
I20260929 17:48:04.660239 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:04.660537 3807 external_mini_cluster.cc:1428] Setting key 294442c142f320325bdc3ca36c15dae8
I20260929 17:48:04.661404 9316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3: Bootstrap starting.
I20260929 17:48:04.663993 9316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:04.664842 9316 log.cc:824] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:04.666924 9316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3: No bootstrap required, opened a new log
I20260929 17:48:04.670607 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:04.662014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.3.183.254:42499)
0929 17:48:04.662813 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:04.662830 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:04.662866 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:48:04.663518 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:04.663536 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:04.664005 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:04.664337 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:04.664724 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.665758 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.666897 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.667569 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.667750 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.668799 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:04.668807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:04.668824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:04.669306 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:04.669367 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:04.669387 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:04.669484 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:04.669603 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 17:48:04.669869 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":146,"threads_started":1}
I20260929 17:48:04.671077 9316 raft_consensus.cc:348] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41a8724f54e45d09a132f0b567927b3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42499 } }
I20260929 17:48:04.671306 9316 raft_consensus.cc:374] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:04.671372 9316 raft_consensus.cc:729] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41a8724f54e45d09a132f0b567927b3, State: Initialized, Role: FOLLOWER
I20260929 17:48:04.671851 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41a8724f54e45d09a132f0b567927b3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42499 } }
I20260929 17:48:04.672006 9316 raft_consensus.cc:388] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:04.672063 9316 raft_consensus.cc:482] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:04.672160 9316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:04.673406 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:48:04.736388 9316 raft_consensus.cc:504] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41a8724f54e45d09a132f0b567927b3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42499 } }
I20260929 17:48:04.736930 9316 leader_election.cc:302] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c41a8724f54e45d09a132f0b567927b3; no voters:
I20260929 17:48:04.737360 9316 leader_election.cc:288] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:04.737488 9321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:04.737758 9321 raft_consensus.cc:686] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [term 1 LEADER]: Becoming Leader. State: Replica: c41a8724f54e45d09a132f0b567927b3, State: Running, Role: LEADER
I20260929 17:48:04.738103 9321 consensus_queue.cc:237] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41a8724f54e45d09a132f0b567927b3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42499 } }
I20260929 17:48:04.738821 9316 sys_catalog.cc:564] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:04.739668 9322 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c41a8724f54e45d09a132f0b567927b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41a8724f54e45d09a132f0b567927b3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42499 } } }
I20260929 17:48:04.739833 9322 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:04.740218 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c41a8724f54e45d09a132f0b567927b3. Latest consensus state: current_term: 1 leader_uuid: "c41a8724f54e45d09a132f0b567927b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41a8724f54e45d09a132f0b567927b3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42499 } } }
I20260929 17:48:04.740366 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:04.740422 9330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:04.743762 9330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:04.749424 9330 catalog_manager.cc:1347] Generated new cluster ID: 03edfb24a4fe4a62bc1282519b35146b
I20260929 17:48:04.749504 9330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:04.764000 9330 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:04.764931 9330 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:04.777678 9330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c41a8724f54e45d09a132f0b567927b3: Generated new TSK 0
I20260929 17:48:04.778477 9330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:48:04.841349 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:04.798241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.3.183.254:42499)
0929 17:48:04.798374 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:04.798379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:04.798496 (+ 117us) server_negotiation.cc:407] Connection header received
0929 17:48:04.798652 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:04.798656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:04.798713 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:04.798831 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:48:04.799080 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.799937 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.800831 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.801056 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.801154 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.801415 (+ 261us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:48:04.840881 (+ 39466us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 17:48:04.841206 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 17:48:04.841436 9319 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 17:48:04.841492 9345 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:04.798142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:42499 (local address 127.0.0.1:45746)
0929 17:48:04.798422 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:04.798434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:04.798527 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:04.798860 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:04.798863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:04.798871 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:48:04.798977 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:04.798983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.800044 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.800048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:04.800718 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:04.800727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:04.801162 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:04.801166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:04.801241 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:04.801263 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:48:04.841135 (+ 39872us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 17:48:04.841315 (+ 180us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.183.254:42499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260929 17:48:04.842178 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9249
2026-09-29T17:48:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (395 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:05 dist-test-slave-wbpt krb5kdc[9357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:05 dist-test-slave-wbpt krb5kdc[9357](info): set up 2 sockets
Sep 29 17:48:05 dist-test-slave-wbpt krb5kdc[9357](info): commencing operation
krb5kdc: starting...
W20260929 17:48:06.359814 3807 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:06 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:48:06.475220 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38443
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:38443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:06.580300 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:06.580547 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:06.580585 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:06.584460 9370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:06.584509 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:06.584527 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:06.584542 9370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:06.584561 9370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:06.589614 9370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:38443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:06.590598 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:06.591526 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:48:06.599831 9370 server_base.cc:1034] running on GCE node
W20260929 17:48:06.599772 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:06.599880 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:06.599761 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:06.600513 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:06.601572 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:06.602732 9370 hybrid_clock.cc:648] HybridClock initialized: now 1790704086602710 us; error 37 us; skew 500 ppm
Sep 29 17:48:06 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704086, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:06.605386 9370 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:48:06.606257 9370 webserver.cc:460] Webserver started at http://127.3.183.254:40857/ using document root <none> and password file <none>
I20260929 17:48:06.606769 9370 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:06.606834 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:06.607045 9370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:06.609072 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "7699beb795a7409198ba9c3656ca7b6c"
format_stamp: "Formatted at 2026-09-29 17:48:06 on dist-test-slave-wbpt"
server_key: "a7cf71dfa84d97ae55f3c6ca5268331d"
server_key_iv: "b31ddb26b631720a764901049f6115b2"
server_key_version: "encryptionkey@0"
I20260929 17:48:06.609637 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "7699beb795a7409198ba9c3656ca7b6c"
format_stamp: "Formatted at 2026-09-29 17:48:06 on dist-test-slave-wbpt"
server_key: "a7cf71dfa84d97ae55f3c6ca5268331d"
server_key_iv: "b31ddb26b631720a764901049f6115b2"
server_key_version: "encryptionkey@0"
I20260929 17:48:06.613266 9370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:48:06.615676 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:06.616883 9370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:48:06.616997 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "7699beb795a7409198ba9c3656ca7b6c"
format_stamp: "Formatted at 2026-09-29 17:48:06 on dist-test-slave-wbpt"
server_key: "a7cf71dfa84d97ae55f3c6ca5268331d"
server_key_iv: "b31ddb26b631720a764901049f6115b2"
server_key_version: "encryptionkey@0"
I20260929 17:48:06.617100 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:06.632372 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:06.634873 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:06.635040 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:06.641523 9370 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:38443
I20260929 17:48:06.641536 9437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:38443 every 8 connection(s)
I20260929 17:48:06.642396 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:06.644937 9438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:06.650142 9438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c: Bootstrap starting.
I20260929 17:48:06.650960 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9370
I20260929 17:48:06.651090 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:06.651389 3807 external_mini_cluster.cc:1428] Setting key 8de55bf58267bd847fd9ece078421937
I20260929 17:48:06.652908 9438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:06.653594 9438 log.cc:824] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:06.655625 9438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c: No bootstrap required, opened a new log
I20260929 17:48:06.659134 9438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7699beb795a7409198ba9c3656ca7b6c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38443 } }
I20260929 17:48:06.659404 9438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:06.659461 9438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7699beb795a7409198ba9c3656ca7b6c, State: Initialized, Role: FOLLOWER
Sep 29 17:48:06 dist-test-slave-wbpt krb5kdc[9357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:06.659960 9438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7699beb795a7409198ba9c3656ca7b6c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38443 } }
I20260929 17:48:06.660092 9438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:06.660162 9438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:06.660259 9438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:06.661235 9438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7699beb795a7409198ba9c3656ca7b6c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38443 } }
I20260929 17:48:06.661587 9438 leader_election.cc:302] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7699beb795a7409198ba9c3656ca7b6c; no voters:
I20260929 17:48:06.661892 9438 leader_election.cc:288] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:06.661978 9443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:06.662132 9443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [term 1 LEADER]: Becoming Leader. State: Replica: 7699beb795a7409198ba9c3656ca7b6c, State: Running, Role: LEADER
I20260929 17:48:06.662415 9443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7699beb795a7409198ba9c3656ca7b6c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38443 } }
I20260929 17:48:06.663146 9438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:06.663544 9445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7699beb795a7409198ba9c3656ca7b6c. Latest consensus state: current_term: 1 leader_uuid: "7699beb795a7409198ba9c3656ca7b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7699beb795a7409198ba9c3656ca7b6c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38443 } } }
I20260929 17:48:06.663666 9445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:06.664003 9444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7699beb795a7409198ba9c3656ca7b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7699beb795a7409198ba9c3656ca7b6c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38443 } } }
I20260929 17:48:06.664115 9444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:06.665683 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:06.652502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.3.183.254:38443)
0929 17:48:06.652891 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:06.652905 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:06.652940 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:48:06.653572 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:06.653597 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:06.653968 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:06.654318 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:06.655180 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:06.656280 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:06.657138 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:06.657627 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:06.657769 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:06.660373 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:06.660378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:06.660391 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:06.660417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:06.662032 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:06.662450 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:06.662456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:06.662462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:06.662680 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:06.662957 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:06.662959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:06.662960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:06.663357 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:06.664234 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:06.664380 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 17:48:06.664677 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
W20260929 17:48:06.667524 9458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:48:06.667594 9458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:48:06.667658 9448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:06.668395 9448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:06.673190 9448 catalog_manager.cc:1347] Generated new cluster ID: 2f787addf3464bd0beb4b2cf33f6086d
I20260929 17:48:06.673255 9448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:06.687829 9448 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:06.688791 9448 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:06.694994 9448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7699beb795a7409198ba9c3656ca7b6c: Generated new TSK 0
I20260929 17:48:06.695640 9448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:48:06.725206 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38443
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:06.867753 9464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:06.868044 9464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:06.868091 9464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:06.872058 9464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:06.872112 9464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:06.872174 9464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:06.877384 9464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:06.878438 9464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:06.879455 9464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:48:06.889443 9464 server_base.cc:1034] running on GCE node
W20260929 17:48:06.889379 9469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:06.889369 9470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:06.889365 9472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:06.889999 9464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:06.890607 9464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:06.891757 9464 hybrid_clock.cc:648] HybridClock initialized: now 1790704086891739 us; error 30 us; skew 500 ppm
Sep 29 17:48:06 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704086, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:06.894356 9464 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:48:06.895164 9464 webserver.cc:460] Webserver started at http://127.3.183.193:44723/ using document root <none> and password file <none>
I20260929 17:48:06.895680 9464 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:06.895738 9464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:06.895941 9464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:06.897933 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "60a29c4a1265445389c951e1f38f2109"
format_stamp: "Formatted at 2026-09-29 17:48:06 on dist-test-slave-wbpt"
server_key: "3382d62ad057d0105f5987629af9e80a"
server_key_iv: "c1545899c99d5cdef4e62fcffb83a968"
server_key_version: "encryptionkey@0"
I20260929 17:48:06.898454 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "60a29c4a1265445389c951e1f38f2109"
format_stamp: "Formatted at 2026-09-29 17:48:06 on dist-test-slave-wbpt"
server_key: "3382d62ad057d0105f5987629af9e80a"
server_key_iv: "c1545899c99d5cdef4e62fcffb83a968"
server_key_version: "encryptionkey@0"
I20260929 17:48:06.902060 9464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:06.904508 9479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:06.905995 9464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:48:06.906092 9464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "60a29c4a1265445389c951e1f38f2109"
format_stamp: "Formatted at 2026-09-29 17:48:06 on dist-test-slave-wbpt"
server_key: "3382d62ad057d0105f5987629af9e80a"
server_key_iv: "c1545899c99d5cdef4e62fcffb83a968"
server_key_version: "encryptionkey@0"
I20260929 17:48:06.906203 9464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:06.920357 9464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:06.923377 9464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:06.923547 9464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:06.924114 9464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:06.924872 9464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:06.924916 9464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:06.924957 9464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:06.924985 9464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:06.933369 9464 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:46553
I20260929 17:48:06.933398 9592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:46553 every 8 connection(s)
I20260929 17:48:06.934244 9464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:06.942000 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9464
I20260929 17:48:06.942152 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:06.942478 3807 external_mini_cluster.cc:1428] Setting key 19a8fc00fa7dfa3a7573ad48b0d3c220
Sep 29 17:48:06 dist-test-slave-wbpt krb5kdc[9357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704086, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:06.948105 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:06.936153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:41747 (local address 127.3.183.254:38443)
0929 17:48:06.936307 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:06.936310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:06.937195 (+ 885us) server_negotiation.cc:407] Connection header received
0929 17:48:06.938520 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:06.938523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:06.938570 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:06.938650 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:06.939880 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:06.940416 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:06.941510 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:06.941760 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:06.941833 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:06.944604 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:06.944607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:06.944609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:06.944632 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:06.946080 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:06.946543 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:06.946546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:06.946548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:06.946636 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:06.946980 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:06.946982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:06.946983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:06.947160 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:06.947232 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:06.947855 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 17:48:06.947981 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:48:06.949224 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:06.936536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38443 (local address 127.3.183.193:41747)
0929 17:48:06.937022 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:06.937072 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:06.938331 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:06.938807 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:06.938814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:06.939174 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:06.939697 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:06.939711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:06.940515 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:06.940518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:06.941419 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:06.941429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:06.941930 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:06.941935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:06.942137 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:06.942781 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:06.942800 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:06.944381 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:06.946190 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:06.946195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:06.946207 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:06.946455 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:06.946721 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:06.946724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:06.946726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:06.946897 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:06.947245 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:06.947251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:06.947541 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 17:48:06.947763 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 17:48:06.948132 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
I20260929 17:48:06.950431 9593 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38443
I20260929 17:48:06.950668 9593 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:06.951113 9593 heartbeater.cc:511] Master 127.3.183.254:38443 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:48:06.952548 9402 ts_manager.cc:194] Registered new tserver with Master: 60a29c4a1265445389c951e1f38f2109 (127.3.183.193:46553)
I20260929 17:48:06.953640 9402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:41747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:48:06.971280 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38443
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:07.083035 9598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:07.083310 9598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:07.083369 9598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:07.087335 9598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:07.087393 9598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:07.087476 9598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:07.092566 9598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:07.093566 9598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:07.094543 9598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:07.103797 9604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:07.103802 9603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:07.103881 9598 server_base.cc:1034] running on GCE node
W20260929 17:48:07.103801 9606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:07.104486 9598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:07.105139 9598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:07.106318 9598 hybrid_clock.cc:648] HybridClock initialized: now 1790704087106301 us; error 27 us; skew 500 ppm
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.109025 9598 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:48:07.109853 9598 webserver.cc:460] Webserver started at http://127.3.183.194:41297/ using document root <none> and password file <none>
I20260929 17:48:07.110399 9598 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:07.110462 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:07.110674 9598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:07.112699 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "0bb26c6facbd480c8ac56f5c7ac3c6f6"
format_stamp: "Formatted at 2026-09-29 17:48:07 on dist-test-slave-wbpt"
server_key: "87b8b6e64cd8fc37c8700e90f1f195e6"
server_key_iv: "850f6dd0b2dfe19e77a03f60a4bd97f0"
server_key_version: "encryptionkey@0"
I20260929 17:48:07.113219 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "0bb26c6facbd480c8ac56f5c7ac3c6f6"
format_stamp: "Formatted at 2026-09-29 17:48:07 on dist-test-slave-wbpt"
server_key: "87b8b6e64cd8fc37c8700e90f1f195e6"
server_key_iv: "850f6dd0b2dfe19e77a03f60a4bd97f0"
server_key_version: "encryptionkey@0"
I20260929 17:48:07.117075 9598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 17:48:07.119603 9613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:07.120913 9598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:48:07.121026 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "0bb26c6facbd480c8ac56f5c7ac3c6f6"
format_stamp: "Formatted at 2026-09-29 17:48:07 on dist-test-slave-wbpt"
server_key: "87b8b6e64cd8fc37c8700e90f1f195e6"
server_key_iv: "850f6dd0b2dfe19e77a03f60a4bd97f0"
server_key_version: "encryptionkey@0"
I20260929 17:48:07.121119 9598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:07.138187 9598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:07.141176 9598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:07.141359 9598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:07.141961 9598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:07.142794 9598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:07.142844 9598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:07.142899 9598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:07.142927 9598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:07.151625 9598 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:37773
I20260929 17:48:07.151656 9726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:37773 every 8 connection(s)
I20260929 17:48:07.152572 9598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:07.157598 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9598
I20260929 17:48:07.157723 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:07.158037 3807 external_mini_cluster.cc:1428] Setting key ad929ccc66f2d61de25a24badbdbbfcc
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:07.167517 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:07.154599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:43461 (local address 127.3.183.254:38443)
0929 17:48:07.154729 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.154733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.155575 (+ 842us) server_negotiation.cc:407] Connection header received
0929 17:48:07.156964 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.156967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.157024 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.157118 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.158630 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.159157 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.160689 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.160933 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.161029 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:07.163507 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:07.163512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:07.163515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:07.163568 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:07.165427 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.165934 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.165939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.165940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.166044 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.166344 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.166347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.166349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.166612 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:07.166727 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:07.167199 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 17:48:07.167326 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:48:07.168321 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:07.154994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38443 (local address 127.3.183.194:43461)
0929 17:48:07.155436 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:07.155478 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:07.156759 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:07.157308 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:07.157316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:07.157813 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:07.158484 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:07.158501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.159276 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.159279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:07.160536 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:07.160548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.161088 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.161092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:07.161305 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:07.161872 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:07.161890 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:07.163297 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:07.165548 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:07.165553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:07.165566 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:07.165826 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:07.166132 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:07.166134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:07.166135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:07.166260 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:07.166712 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:07.166717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:07.166954 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 17:48:07.167117 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 17:48:07.167379 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260929 17:48:07.169545 9727 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38443
I20260929 17:48:07.169875 9727 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:48:07.170418 9727 heartbeater.cc:511] Master 127.3.183.254:38443 requested a full tablet report, sending...
I20260929 17:48:07.171479 9402 ts_manager.cc:194] Registered new tserver with Master: 0bb26c6facbd480c8ac56f5c7ac3c6f6 (127.3.183.194:37773)
I20260929 17:48:07.172158 9402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:43461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:48:07.186179 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38443
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:07.293100 9732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:07.293339 9732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:07.293395 9732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:07.297370 9732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:07.297428 9732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:07.297509 9732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:07.302742 9732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:07.303736 9732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:07.304738 9732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:07.313984 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:07.313961 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:07.314002 9732 server_base.cc:1034] running on GCE node
W20260929 17:48:07.313985 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:07.314550 9732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:07.315169 9732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:07.316339 9732 hybrid_clock.cc:648] HybridClock initialized: now 1790704087316308 us; error 39 us; skew 500 ppm
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.319139 9732 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:48:07.320045 9732 webserver.cc:460] Webserver started at http://127.3.183.195:33559/ using document root <none> and password file <none>
I20260929 17:48:07.320580 9732 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:07.320640 9732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:07.320803 9732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:07.322829 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "7c5ede15580340c7aea211fd3896dd69"
format_stamp: "Formatted at 2026-09-29 17:48:07 on dist-test-slave-wbpt"
server_key: "ac9caf67c491ab16c174daa3074ffdcb"
server_key_iv: "65868f08468b9522592254c4972d4970"
server_key_version: "encryptionkey@0"
I20260929 17:48:07.323354 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "7c5ede15580340c7aea211fd3896dd69"
format_stamp: "Formatted at 2026-09-29 17:48:07 on dist-test-slave-wbpt"
server_key: "ac9caf67c491ab16c174daa3074ffdcb"
server_key_iv: "65868f08468b9522592254c4972d4970"
server_key_version: "encryptionkey@0"
I20260929 17:48:07.327867 9732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 17:48:07.330490 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:07.331730 9732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:07.331862 9732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "7c5ede15580340c7aea211fd3896dd69"
format_stamp: "Formatted at 2026-09-29 17:48:07 on dist-test-slave-wbpt"
server_key: "ac9caf67c491ab16c174daa3074ffdcb"
server_key_iv: "65868f08468b9522592254c4972d4970"
server_key_version: "encryptionkey@0"
I20260929 17:48:07.331979 9732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:07.357226 9732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:07.360549 9732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:07.360737 9732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:07.361344 9732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:07.362239 9732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:07.362293 9732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:07.362344 9732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:07.362377 9732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:07.371302 9732 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:36711
I20260929 17:48:07.371322 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:36711 every 8 connection(s)
I20260929 17:48:07.372227 9732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:07.382249 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9732
I20260929 17:48:07.382367 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:07.382752 3807 external_mini_cluster.cc:1428] Setting key 86b6854deebb813ceb5ef0892d65d7e1
I20260929 17:48:07.385082 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:07.373909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:49069 (local address 127.3.183.254:38443)
0929 17:48:07.374049 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.374054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.374896 (+ 842us) server_negotiation.cc:407] Connection header received
0929 17:48:07.376223 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.376226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.376274 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.376355 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.377441 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.377981 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.379048 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.379267 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.379354 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:07.381571 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:07.381574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:07.381576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:07.381599 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:07.383074 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.383553 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.383556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.383558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.383657 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.384068 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.384072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.384074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.384272 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:07.384365 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:07.384842 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 17:48:07.384961 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:48:07.385811 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:07.374324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38443 (local address 127.3.183.195:49069)
0929 17:48:07.374757 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:07.374797 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:07.376008 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:07.376493 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:07.376500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:07.376824 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:07.377308 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:07.377322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.378084 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.378087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:07.378946 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:07.378954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.379387 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.379390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:07.379517 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:07.380037 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:07.380052 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:07.381428 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:07.383179 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:07.383184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:07.383197 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:07.383446 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:07.383755 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:07.383758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:07.383760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:07.383984 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:07.384369 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:07.384374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:07.384583 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 17:48:07.384755 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 17:48:07.385039 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20260929 17:48:07.386790 9861 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38443
I20260929 17:48:07.386993 9861 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:07.387459 9861 heartbeater.cc:511] Master 127.3.183.254:38443 requested a full tablet report, sending...
I20260929 17:48:07.388384 9402 ts_manager.cc:194] Registered new tserver with Master: 7c5ede15580340c7aea211fd3896dd69 (127.3.183.195:36711)
I20260929 17:48:07.388883 9402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:49069
I20260929 17:48:07.396456 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/test-admin.keytab.
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.412410 3807 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 17:48:07.415032 9868 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:07.413643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.3.183.193:46553)
0929 17:48:07.413906 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.413922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.413947 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:48:07.414083 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.414089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.414279 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.414406 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.414863 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38654: BlockingRecv error: recv got EOF from 127.0.0.1:38654 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
W20260929 17:48:07.415148 3807 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5cc' of type FILE
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.416154 3807 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.417042 3807 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:48:07.426222 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:07.417832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.3.183.193:46553)
0929 17:48:07.417928 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.417931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.418125 (+ 194us) server_negotiation.cc:407] Connection header received
0929 17:48:07.418195 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.418198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.418239 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.418318 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.418785 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.419615 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.420499 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.420859 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.420959 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:07.423031 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:07.423040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:07.423053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:07.423076 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:07.424641 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.425051 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.425057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.425058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.425135 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.425336 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.425338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.425339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.425617 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:07.425771 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:07.425923 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 17:48:07.426067 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 17:48:07.429816 9868 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:07.428898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.3.183.193:46553)
0929 17:48:07.429030 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.429033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.429079 (+ 46us) server_negotiation.cc:407] Connection header received
0929 17:48:07.429183 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.429185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.429218 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.429273 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.429737 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38686: BlockingRecv error: recv got EOF from 127.0.0.1:38686 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 17:48:07.429910 3807 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5cc' of type FILE
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.430926 3807 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.431808 3807 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:48:07.439512 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:07.432517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.3.183.193:46553)
0929 17:48:07.432617 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.432620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.432721 (+ 101us) server_negotiation.cc:407] Connection header received
0929 17:48:07.432798 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.432800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.432832 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.432891 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.433306 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.433810 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.434565 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.434797 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.434887 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:07.436986 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:07.436989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:07.436991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:07.437011 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:07.438235 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.438587 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.438590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.438591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.438675 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.438904 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.438907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.438908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.439062 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:07.439168 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:07.439305 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 17:48:07.439396 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 17:48:07.441864 9868 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:07.441011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.3.183.193:46553)
0929 17:48:07.441116 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.441120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.441188 (+ 68us) server_negotiation.cc:407] Connection header received
0929 17:48:07.441264 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.441266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.441297 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.441355 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.441779 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38696: BlockingRecv error: recv got EOF from 127.0.0.1:38696 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 17:48:07.441975 3807 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704058755242-3807-0/krb5kdc/krb5cc' of type FILE
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.442889 3807 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:07.443698 3807 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:48:07.451293 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:07.444481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.3.183.193:46553)
0929 17:48:07.444563 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:07.444566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:07.444659 (+ 93us) server_negotiation.cc:407] Connection header received
0929 17:48:07.444719 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:07.444721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:07.444751 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:07.444806 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:07.445220 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.445687 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.446426 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:07.446631 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:07.446705 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:07.448689 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:07.448693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:07.448694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:07.448713 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:07.449908 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.450285 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.450288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.450289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.450353 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:07.450594 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:07.450597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:07.450599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:07.450751 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:07.450854 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:07.451085 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 17:48:07.451184 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 17:48:07.452113 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9464
I20260929 17:48:07.457219 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9598
I20260929 17:48:07.461591 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9732
I20260929 17:48:07.465852 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9370
2026-09-29T17:48:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2386 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9881](info): set up 2 sockets
Sep 29 17:48:07 dist-test-slave-wbpt krb5kdc[9881](info): commencing operation
krb5kdc: starting...
W20260929 17:48:08.419919 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:08 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab.
Entry for principal oryx/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab.
I20260929 17:48:08.536852 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:45769
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab
--principal=oryx/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:45769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:08.644371 9894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:08.644639 9894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:08.644686 9894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:08.648684 9894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:08.648756 9894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:08.648777 9894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:08.648797 9894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:08.648826 9894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:08.654263 9894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:45769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab
--principal=oryx/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:45769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:08.655360 9894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:08.656427 9894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:08.664801 9900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:08.664876 9902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:08.664804 9899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:08.665019 9894 server_base.cc:1034] running on GCE node
I20260929 17:48:08.665755 9894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:08.666929 9894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:08.668115 9894 hybrid_clock.cc:648] HybridClock initialized: now 1790704088668085 us; error 42 us; skew 500 ppm
Sep 29 17:48:08 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:08.670703 9894 init.cc:377] Logged in from keytab as oryx/127.3.183.254@KRBTEST.COM (short username oryx)
I20260929 17:48:08.671566 9894 webserver.cc:460] Webserver started at http://127.3.183.254:40455/ using document root <none> and password file <none>
I20260929 17:48:08.672155 9894 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:08.672236 9894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:08.672435 9894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:08.674408 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "8b694f0ceec04751a088dc6d859267bb"
format_stamp: "Formatted at 2026-09-29 17:48:08 on dist-test-slave-wbpt"
server_key: "3954a326031e97b3275e7338604f3847"
server_key_iv: "a0518a8ff097862036b1d52c0c4f9081"
server_key_version: "encryptionkey@0"
I20260929 17:48:08.674928 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "8b694f0ceec04751a088dc6d859267bb"
format_stamp: "Formatted at 2026-09-29 17:48:08 on dist-test-slave-wbpt"
server_key: "3954a326031e97b3275e7338604f3847"
server_key_iv: "a0518a8ff097862036b1d52c0c4f9081"
server_key_version: "encryptionkey@0"
I20260929 17:48:08.678561 9894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:48:08.680855 9909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:08.682046 9894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:08.682175 9894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "8b694f0ceec04751a088dc6d859267bb"
format_stamp: "Formatted at 2026-09-29 17:48:08 on dist-test-slave-wbpt"
server_key: "3954a326031e97b3275e7338604f3847"
server_key_iv: "a0518a8ff097862036b1d52c0c4f9081"
server_key_version: "encryptionkey@0"
I20260929 17:48:08.682269 9894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:08.729266 9894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:08.732168 9894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:08.732339 9894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:08.738941 9894 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:45769
I20260929 17:48:08.739243 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:45769 every 8 connection(s)
I20260929 17:48:08.739951 9894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:08.742662 9962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:08.742664 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9894
I20260929 17:48:08.742774 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:08.743126 3807 external_mini_cluster.cc:1428] Setting key 137e890c2934bd990d7459124a65126d
I20260929 17:48:08.749328 9962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: Bootstrap starting.
I20260929 17:48:08.752000 9962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: Neither blocks nor log segments found. Creating new log.
Sep 29 17:48:08 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:08.752697 9962 log.cc:824] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:08.754906 9962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: No bootstrap required, opened a new log
I20260929 17:48:08.757416 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:08.744589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.3.183.254:45769)
0929 17:48:08.744974 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:08.744992 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:08.745026 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:48:08.745819 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:08.745836 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:08.746158 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:08.746421 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:08.747253 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:08.748305 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:08.749392 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:08.749885 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:08.749994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:08.752548 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:08.752553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:08.752566 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:08.752592 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:08.754388 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:08.754797 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:08.754803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:08.754809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:08.754947 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:08.755205 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:08.755208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:08.755210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:08.755697 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:08.755920 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:08.756132 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 17:48:08.756422 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260929 17:48:08.759189 9962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } }
I20260929 17:48:08.759420 9962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:08.759502 9962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b694f0ceec04751a088dc6d859267bb, State: Initialized, Role: FOLLOWER
I20260929 17:48:08.759950 9962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } }
I20260929 17:48:08.760074 9962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:08.760134 9962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:08.760224 9962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:08.761052 9962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } }
I20260929 17:48:08.761358 9962 leader_election.cc:302] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b694f0ceec04751a088dc6d859267bb; no voters:
I20260929 17:48:08.761628 9962 leader_election.cc:288] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:08.761713 9967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:08.761906 9967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 1 LEADER]: Becoming Leader. State: Replica: 8b694f0ceec04751a088dc6d859267bb, State: Running, Role: LEADER
I20260929 17:48:08.762252 9967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } }
I20260929 17:48:08.762832 9962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:08.763880 9969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b694f0ceec04751a088dc6d859267bb. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } }
I20260929 17:48:08.763888 9968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } }
I20260929 17:48:08.764004 9969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:08.764004 9968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:08.764384 9976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:08.767283 9976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:08.774554 9976 catalog_manager.cc:1347] Generated new cluster ID: 4cc877435e54456b945bc782e7d97106
I20260929 17:48:08.774642 9976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:08.785585 9976 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:08.786886 9976 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:08.799037 9976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: Generated new TSK 0
I20260929 17:48:08.799829 9976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab.
Entry for principal oryx/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab.
I20260929 17:48:08.832078 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45769
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab
--principal=oryx/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:08.946462 9988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:08.946713 9988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:08.946766 9988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:08.950696 9988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:08.950752 9988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:08.950809 9988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:08.955876 9988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab
--principal=oryx/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:08.956833 9988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:08.957828 9988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:08.966986 9996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:08.967020 9994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:08.967178 9988 server_base.cc:1034] running on GCE node
W20260929 17:48:08.967022 9993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:08.967633 9988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:08.968272 9988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:08.969439 9988 hybrid_clock.cc:648] HybridClock initialized: now 1790704088969409 us; error 39 us; skew 500 ppm
Sep 29 17:48:08 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:08.972265 9988 init.cc:377] Logged in from keytab as oryx/127.3.183.193@KRBTEST.COM (short username oryx)
I20260929 17:48:08.973079 9988 webserver.cc:460] Webserver started at http://127.3.183.193:32975/ using document root <none> and password file <none>
I20260929 17:48:08.973587 9988 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:08.973650 9988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:08.973834 9988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:08.975817 9988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "a7b7278a49834eba9417ddb666333110"
format_stamp: "Formatted at 2026-09-29 17:48:08 on dist-test-slave-wbpt"
server_key: "3ef7832a2f47129b7159ed39016e5571"
server_key_iv: "4411a9059f4c1daa4ce365f57a372350"
server_key_version: "encryptionkey@0"
I20260929 17:48:08.976346 9988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "a7b7278a49834eba9417ddb666333110"
format_stamp: "Formatted at 2026-09-29 17:48:08 on dist-test-slave-wbpt"
server_key: "3ef7832a2f47129b7159ed39016e5571"
server_key_iv: "4411a9059f4c1daa4ce365f57a372350"
server_key_version: "encryptionkey@0"
I20260929 17:48:08.979928 9988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:48:08.982303 10003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:08.983528 9988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:08.983633 9988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "a7b7278a49834eba9417ddb666333110"
format_stamp: "Formatted at 2026-09-29 17:48:08 on dist-test-slave-wbpt"
server_key: "3ef7832a2f47129b7159ed39016e5571"
server_key_iv: "4411a9059f4c1daa4ce365f57a372350"
server_key_version: "encryptionkey@0"
I20260929 17:48:08.983731 9988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:08.999707 9988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:09.002789 9988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:09.002965 9988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:09.003515 9988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:09.004315 9988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:09.004362 9988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.004415 9988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:09.004443 9988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.012712 9988 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:39299
I20260929 17:48:09.012743 10116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:39299 every 8 connection(s)
I20260929 17:48:09.013607 9988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:09.017967 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 9988
I20260929 17:48:09.018106 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:09.018409 3807 external_mini_cluster.cc:1428] Setting key 14dda900056d38b15b73c7132b447f5b
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:48:09.027693 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.015315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:51083 (local address 127.3.183.254:45769)
0929 17:48:09.015442 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.015446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.016525 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 17:48:09.017360 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.017363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.017408 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.017477 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.018951 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.019653 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.021205 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.021450 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.021540 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.023914 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.023918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.023921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.023950 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.025543 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.026089 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.026093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.026095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.026189 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.026556 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.026559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.026561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.026797 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.026899 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.027402 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.027533 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:48:09.028590 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.015704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:51083)
0929 17:48:09.016386 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.016427 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.017194 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.017628 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.017634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.018141 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.018801 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.018819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.019803 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.019807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.021077 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.021089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.021606 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.021610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.021768 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.022296 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.022312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.023719 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.025673 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.025678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.025691 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.025977 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.026300 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.026302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.026304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.026453 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.026908 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.026913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.027154 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.027315 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.027621 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":112,"threads_started":1}
I20260929 17:48:09.029812 10117 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.194@KRBTEST.COM" created.
I20260929 17:48:09.030049 10117 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:09.030625 10117 heartbeater.cc:511] Master 127.3.183.254:45769 requested a full tablet report, sending...
I20260929 17:48:09.032351 9926 ts_manager.cc:194] Registered new tserver with Master: a7b7278a49834eba9417ddb666333110 (127.3.183.193:39299)
I20260929 17:48:09.033506 9926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:51083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab.
Entry for principal oryx/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab.
I20260929 17:48:09.046774 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45769
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab
--principal=oryx/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:09.156180 10122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:09.156450 10122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:09.156493 10122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:09.160457 10122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:09.160514 10122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:09.160569 10122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:09.165936 10122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab
--principal=oryx/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:09.166879 10122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:09.167840 10122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:09.178479 10127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.178474 10128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.178494 10130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:09.178880 10122 server_base.cc:1034] running on GCE node
I20260929 17:48:09.179250 10122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:09.179929 10122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:09.181109 10122 hybrid_clock.cc:648] HybridClock initialized: now 1790704089181076 us; error 38 us; skew 500 ppm
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:09.183738 10122 init.cc:377] Logged in from keytab as oryx/127.3.183.194@KRBTEST.COM (short username oryx)
I20260929 17:48:09.184626 10122 webserver.cc:460] Webserver started at http://127.3.183.194:34253/ using document root <none> and password file <none>
I20260929 17:48:09.185168 10122 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:09.185228 10122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:09.185411 10122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:09.187565 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "d7f0d93473db461695d14be89edf0640"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "203d90607ac66e11c8e1597f9b8ad369"
server_key_iv: "b8e7b7f3b3f7a4a5fe0b775fc7f649a4"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.188105 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "d7f0d93473db461695d14be89edf0640"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "203d90607ac66e11c8e1597f9b8ad369"
server_key_iv: "b8e7b7f3b3f7a4a5fe0b775fc7f649a4"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.192281 10122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:48:09.194696 10137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.195987 10122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:09.196091 10122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "d7f0d93473db461695d14be89edf0640"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "203d90607ac66e11c8e1597f9b8ad369"
server_key_iv: "b8e7b7f3b3f7a4a5fe0b775fc7f649a4"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.196203 10122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:09.216364 10122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:09.219470 10122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:09.219647 10122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:09.220376 10122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:09.221259 10122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:09.221305 10122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.221342 10122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:09.221361 10122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.229416 10122 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:37975
I20260929 17:48:09.229454 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:37975 every 8 connection(s)
I20260929 17:48:09.230255 10122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:09.233457 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 10122
I20260929 17:48:09.233562 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:09.233819 3807 external_mini_cluster.cc:1428] Setting key 0a17ba4a50ec443be2cb7355b1a0f943
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:48:09.243827 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.232067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:50403 (local address 127.3.183.254:45769)
0929 17:48:09.232214 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.232218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.233198 (+ 980us) server_negotiation.cc:407] Connection header received
0929 17:48:09.234117 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.234121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.234178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.234251 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.235557 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.236115 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.237603 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.237858 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.237957 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.240533 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.240536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.240538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.240561 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.241844 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.242388 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.242390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.242391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.242456 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.242783 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.242785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.242786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.242948 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.243036 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.243578 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.243681 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:48:09.244683 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.232444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:50403)
0929 17:48:09.233056 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.233095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.233920 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.234435 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.234442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.234914 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.235418 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.235432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.236228 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.236231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.237477 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.237488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.237986 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.237989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.238138 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.238664 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.238680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.240364 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.241954 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.241960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.241975 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.242301 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.242532 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.242535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.242537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.242705 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.243050 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.243057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.243339 (+ 282us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.243544 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.243825 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.195@KRBTEST.COM" created.
I20260929 17:48:09.245873 10251 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45769
I20260929 17:48:09.246115 10251 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:09.246605 10251 heartbeater.cc:511] Master 127.3.183.254:45769 requested a full tablet report, sending...
I20260929 17:48:09.247618 9926 ts_manager.cc:194] Registered new tserver with Master: d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975)
I20260929 17:48:09.248386 9926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:50403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab.
Entry for principal oryx/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab.
I20260929 17:48:09.263494 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45769
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab
--principal=oryx/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:09.371925 10256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:09.372157 10256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:09.372198 10256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:09.376085 10256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:09.376132 10256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:09.376187 10256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:09.381197 10256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab
--principal=oryx/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:09.382102 10256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:09.383055 10256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:09.392453 10262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.392472 10264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.392500 10261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:09.392681 10256 server_base.cc:1034] running on GCE node
I20260929 17:48:09.393086 10256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:09.393728 10256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:09.394906 10256 hybrid_clock.cc:648] HybridClock initialized: now 1790704089394878 us; error 41 us; skew 500 ppm
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:09.397631 10256 init.cc:377] Logged in from keytab as oryx/127.3.183.195@KRBTEST.COM (short username oryx)
I20260929 17:48:09.398466 10256 webserver.cc:460] Webserver started at http://127.3.183.195:40417/ using document root <none> and password file <none>
I20260929 17:48:09.398986 10256 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:09.399053 10256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:09.399271 10256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:09.401352 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "a32093fe7d284f5bb9717e285bf918da"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "7615f18dc6552e38f42e5e399889e1a0"
server_key_iv: "1dda1113f51039b4c7a860c959e2e6e7"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.401886 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "a32093fe7d284f5bb9717e285bf918da"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "7615f18dc6552e38f42e5e399889e1a0"
server_key_iv: "1dda1113f51039b4c7a860c959e2e6e7"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.405537 10256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:09.407735 10271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.408902 10256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:09.409015 10256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "a32093fe7d284f5bb9717e285bf918da"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "7615f18dc6552e38f42e5e399889e1a0"
server_key_iv: "1dda1113f51039b4c7a860c959e2e6e7"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.409101 10256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:09.432678 10256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:09.435809 10256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:09.435981 10256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:09.436558 10256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:09.437371 10256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:09.437418 10256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.437457 10256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:09.437474 10256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.445920 10256 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:44667
I20260929 17:48:09.445950 10384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:44667 every 8 connection(s)
I20260929 17:48:09.446777 10256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:09.449474 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 10256
I20260929 17:48:09.449600 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:09.449930 3807 external_mini_cluster.cc:1428] Setting key 5c3fdba7ec7f0412de047413b2a3cb8a
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:09.458930 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.448436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:34821 (local address 127.3.183.254:45769)
0929 17:48:09.448555 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.448559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.449668 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 17:48:09.450625 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.450628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.450685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.450763 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.452123 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.452670 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.453724 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.453939 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.454010 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.456274 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.456277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.456279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.456302 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.457401 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.457808 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.457811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.457812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.457876 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.458108 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.458110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.458111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.458276 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.458345 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.458728 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.458814 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:48:09.459692 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.448804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:34821)
0929 17:48:09.449496 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.449551 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.450451 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.450963 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.450972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.451410 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.452005 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.452021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.452768 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.452770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.453637 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.453646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.454050 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.454053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.454206 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.454750 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.454766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.456116 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.457488 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.457493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.457504 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.457740 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.457934 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.457936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.457937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.458050 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.458343 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.458347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.458555 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.458698 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.458929 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":112,"threads_started":1}
I20260929 17:48:09.460736 10385 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45769
I20260929 17:48:09.460937 10385 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:09.461365 10385 heartbeater.cc:511] Master 127.3.183.254:45769 requested a full tablet report, sending...
I20260929 17:48:09.462319 9926 ts_manager.cc:194] Registered new tserver with Master: a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667)
I20260929 17:48:09.463028 9926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:34821
I20260929 17:48:09.463485 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.183.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.253.keytab.
Entry for principal oryx/127.3.183.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.253.keytab.
I20260929 17:48:09.488701 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:38113
--webserver_interface=127.3.183.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.253.keytab
--principal=oryx/127.3.183.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:45769,127.3.183.253:38113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:09.595124 10390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:09.595397 10390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:09.595438 10390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:09.599474 10390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:09.599552 10390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:09.599577 10390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:09.599601 10390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:09.599619 10390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:09.605110 10390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:45769,127.3.183.253:38113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.253.keytab
--principal=oryx/127.3.183.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:38113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.183.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
W20260929 17:48:09.605343 10390 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.183.254:45769,127.3.183.253:38113'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 17:48:09.606238 10390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:09.607208 10390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:09.615643 10398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.615621 10395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.615926 10396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:09.616165 10390 server_base.cc:1034] running on GCE node
I20260929 17:48:09.616742 10390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:09.617871 10390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:09.619030 10390 hybrid_clock.cc:648] HybridClock initialized: now 1790704089619012 us; error 31 us; skew 500 ppm
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:09.621693 10390 init.cc:377] Logged in from keytab as oryx/127.3.183.253@KRBTEST.COM (short username oryx)
I20260929 17:48:09.622574 10390 webserver.cc:460] Webserver started at http://127.3.183.253:34949/ using document root <none> and password file <none>
I20260929 17:48:09.623090 10390 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:09.623173 10390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:09.623374 10390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:09.625429 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/instance:
uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "51f116b22955ee10cefa514dcdf32c0c"
server_key_iv: "1904b0cdcbbe5a1bdc6496c535673ce3"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.625953 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal/instance:
uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "51f116b22955ee10cefa514dcdf32c0c"
server_key_iv: "1904b0cdcbbe5a1bdc6496c535673ce3"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.629660 10390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 17:48:09.631955 10405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.633230 10390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:48:09.633347 10390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "51f116b22955ee10cefa514dcdf32c0c"
server_key_iv: "1904b0cdcbbe5a1bdc6496c535673ce3"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.633440 10390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:09.645874 10390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:09.648819 10390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:09.648988 10390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:09.655483 10390 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.253:38113
I20260929 17:48:09.655735 10457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.253:38113 every 8 connection(s)
I20260929 17:48:09.656461 10390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
I20260929 17:48:09.659109 10458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:09.662904 10458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:48:09.664244 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 10390
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.253@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.183.253@KRBTEST.COM
I20260929 17:48:09.675863 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.664891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.3.183.254:45769)
0929 17:48:09.665033 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.665037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.666051 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 17:48:09.666745 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.666748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.666796 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.666887 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.668016 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.668515 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.669483 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.669699 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.669788 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.672167 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.672171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.672173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.672199 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.673720 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.674262 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.674266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.674268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.674349 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.674692 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.674695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.674697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.674951 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.675048 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.675554 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.675672 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:48:09.676694 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.665406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60266)
0929 17:48:09.665909 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.665954 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.666534 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.667313 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.667316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.667646 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.667890 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.667900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.668632 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.668636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.669371 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.669379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.669831 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.669835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.669934 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.670536 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.670551 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.672021 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.673817 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.673824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.673833 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.674174 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.674428 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.674431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.674433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.674612 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.675048 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.675053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.675329 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.675490 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.675877 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":31,"spinlock_wait_cycles":3712,"thread_start_us":176,"threads_started":1}
I20260929 17:48:09.676698 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.665405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.3.183.253:38113)
0929 17:48:09.665797 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.665813 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.665852 (+ 39us) server_negotiation.cc:407] Connection header received
0929 17:48:09.666527 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.666545 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.666765 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.666932 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.667571 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.668631 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.669399 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.669759 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.669893 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.672619 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.672624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.672634 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.672659 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.674199 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.674603 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.674607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.674608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.674684 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.674914 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.674917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.674919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.675323 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.675465 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.675693 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.676016 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":228,"server-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260929 17:48:09.678007 10458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.253@KRBTEST.COM for oryx/127.3.183.253@KRBTEST.COM
I20260929 17:48:09.684974 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.678418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.0.0.1:46440)
0929 17:48:09.678522 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.678535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.678603 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.678970 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.678973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.679111 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.679197 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.679201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.679860 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.679863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.680365 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.680371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.680762 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.680766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.680849 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.681208 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.681217 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.682326 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.683657 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.683660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.683662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.683896 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.684116 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.684119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.684121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.684191 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.684516 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.684519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.684595 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.684673 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.684803 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 17:48:09.684974 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.678532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.3.183.253:38113)
0929 17:48:09.678812 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.678816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.678827 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:48:09.678854 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.678857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.678889 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.678948 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.679286 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.679763 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.680455 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.680677 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.680758 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.682433 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.682437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.682440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.682459 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.683570 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.683968 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.683971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.683973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.684045 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.684258 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.684261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.684263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.684432 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.684509 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.684723 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.684852 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":217}
I20260929 17:48:09.688191 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Bootstrap starting.
I20260929 17:48:09.690395 10458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:09.691115 10458 log.cc:824] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:09.693102 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: No bootstrap required, opened a new log
I20260929 17:48:09.696198 10458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } }
I20260929 17:48:09.696405 10458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:09.696460 10458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8390d94a4ac14b62a5abb1b0cf25f68c, State: Initialized, Role: FOLLOWER
I20260929 17:48:09.696867 10458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } }
I20260929 17:48:09.697297 10464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } } }
I20260929 17:48:09.697409 10464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:48:09.698057 10458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 17:48:09.702839 10475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:48:09.702919 10475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:48:09.704562 10390 master_runner.cc:428] Detected that this master 8390d94a4ac14b62a5abb1b0cf25f68c is joining an existing cluster
I20260929 17:48:09.704622 10390 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.183.253:38113
I20260929 17:48:09.715685 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.706927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.3.183.253:38113)
0929 17:48:09.707018 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.707036 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.707175 (+ 139us) server_negotiation.cc:407] Connection header received
0929 17:48:09.707277 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.707279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.707314 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.707377 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.708054 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.708809 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.709971 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.710177 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.710263 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.711387 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.711391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.711394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.711419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.713242 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.714555 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.714559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.714561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.714649 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.714912 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.714916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.714917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.715111 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.715187 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.715402 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.715545 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":27}
I20260929 17:48:09.715699 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.706580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.0.0.1:46446)
0929 17:48:09.707089 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.707101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.707198 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.707406 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.707409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.707726 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.707918 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.707927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.708924 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.708928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.709851 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.709862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.710292 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.710295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.710379 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.710820 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.710834 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.711261 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.713366 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.713370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.713372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.714460 (+ 1088us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.714744 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.714747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.714749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.714822 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.715186 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.715189 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.715270 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.715355 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.715498 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":7,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260929 17:48:09.716341 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.706538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60268)
0929 17:48:09.706928 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.706942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.707051 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.707452 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.707454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.707726 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.707904 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.707910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.709995 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.709998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.710881 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.710891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.711418 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.711422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.711505 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.712480 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.712496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.712853 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.714412 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.714415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.714417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.714629 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.714867 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.714870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.714872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.714954 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.715467 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.715470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.715963 (+ 493us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.716052 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.716173 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":18,"thread_start_us":114,"threads_started":1}
I20260929 17:48:09.716372 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.706620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.3.183.254:45769)
0929 17:48:09.706748 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.706753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.707003 (+ 250us) server_negotiation.cc:407] Connection header received
0929 17:48:09.707139 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.707142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.707192 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.707262 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.708008 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.708810 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.711005 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.711334 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.711424 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.712979 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.712983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.712985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.713012 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.714312 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.714709 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.714712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.714714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.714790 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.715055 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.715058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.715060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.715278 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.715374 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.716132 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.716244 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.254@KRBTEST.COM for oryx/127.3.183.253@KRBTEST.COM
I20260929 17:48:09.729210 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.720165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.3.183.253:38113)
0929 17:48:09.720279 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.720283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.720582 (+ 299us) server_negotiation.cc:407] Connection header received
0929 17:48:09.720766 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.720769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.720817 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.720888 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.721449 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.722201 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.723018 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.723350 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.723439 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.725740 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.725744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.725747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.725775 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.727356 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.727896 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.727899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.727901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.727989 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.728315 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.728318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.728319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.728531 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.728622 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.728917 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.729043 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:48:09.729243 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.720124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.0.0.1:46462)
0929 17:48:09.720463 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.720514 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.720646 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.720925 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.720929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.721260 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.721360 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.721365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.722285 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.722287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.722921 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.722928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.723447 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.723450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.723535 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.724021 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.724032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.725593 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.727484 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.727493 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.727499 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.727754 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.728143 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.728146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.728147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.728233 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.728621 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.728627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.728736 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.728851 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.729056 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
I20260929 17:48:09.730814 9926 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.183.253:38113: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
member_type: NON_VOTER
last_known_addr {
host: "127.3.183.253"
port: 38113
}
attrs {
promote: true
}
}
dest_uuid: "8b694f0ceec04751a088dc6d859267bb"
cas_config_opid_index: -1
I20260929 17:48:09.731536 9926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } }
I20260929 17:48:09.733388 10485 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } }
I20260929 17:48:09.734041 10432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:09.735194 10485 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.183.253:38113
I20260929 17:48:09.735390 10432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Refusing update from remote peer 8b694f0ceec04751a088dc6d859267bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 17:48:09.735504 10485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } }
I20260929 17:48:09.735589 10485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:09.736119 10485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:09.736197 10390 master.cc:537] Master@127.3.183.253:38113 shutting down...
W20260929 17:48:09.738338 9912 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.3.183.252@KRBTEST.COM; defaulting to no policy
W20260929 17:48:09.740162 9912 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb -> Peer 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253:38113): Couldn't send request to peer 8390d94a4ac14b62a5abb1b0cf25f68c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.252@KRBTEST.COM" created.
I20260929 17:48:09.745962 10390 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:09.746273 10390 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:09.746390 10390 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: stopping tablet replica
I20260929 17:48:09.751327 10390 master.cc:552] Master@127.3.183.253:38113 shutdown complete.
I20260929 17:48:09.751376 10390 master_runner.cc:305] Clearing existing system tablet
I20260929 17:48:09.751605 10390 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:09.751657 10390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:09.752925 10390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:09.753891 10491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.754117 10390 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:09.754391 10390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:09.754467 10390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "51f116b22955ee10cefa514dcdf32c0c"
server_key_iv: "1904b0cdcbbe5a1bdc6496c535673ce3"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.756733 10390 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.252.keytab.
Entry for principal oryx/127.3.183.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.252.keytab.
I20260929 17:48:09.759009 10390 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 17:48:09.759079 10390 log.cc:1196] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 17:48:09.759244 10390 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Deleting consensus metadata
I20260929 17:48:09.759338 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:44775
--webserver_interface=127.3.183.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.252.keytab
--principal=oryx/127.3.183.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:48:09.759384 10390 master_runner.cc:315] Copying system tablet from 127.3.183.254:45769
I20260929 17:48:09.760567 10390 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.183.254:45769
I20260929 17:48:09.770164 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.761494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.3.183.254:45769)
0929 17:48:09.761610 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.761614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.761631 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:48:09.761679 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.761682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.761730 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.761818 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.762331 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.762956 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.764161 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.764583 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.764714 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.766119 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.766124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.766127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.766160 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:09.768155 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.768665 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.768669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.768671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.768886 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:09.769200 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:09.769203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:09.769205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:09.769456 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:09.769558 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:09.769850 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 17:48:09.769977 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:48:09.770180 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.761115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60270)
0929 17:48:09.761390 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.761406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.761519 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.761841 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.761844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.762060 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.762176 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.762183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.763075 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.763078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.764041 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.764053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.764816 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.764819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.764915 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.765504 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.765525 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.765959 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:09.768293 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.768297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.768299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.768562 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.768992 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:09.768996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:09.768998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:09.769107 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:09.769556 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:09.769559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:09.769672 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 17:48:09.769787 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 17:48:09.769942 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
I20260929 17:48:09.770750 9946 tablet_copy_service.cc:140] P 8b694f0ceec04751a088dc6d859267bb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8390d94a4ac14b62a5abb1b0cf25f68c ({username='oryx', principal='oryx/127.3.183.253@KRBTEST.COM'} at 127.0.0.1:60270)
I20260929 17:48:09.770953 9946 tablet_copy_service.cc:161] P 8b694f0ceec04751a088dc6d859267bb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8390d94a4ac14b62a5abb1b0cf25f68c at {username='oryx', principal='oryx/127.3.183.253@KRBTEST.COM'} at 127.0.0.1:60270: session id = 8390d94a4ac14b62a5abb1b0cf25f68c-00000000000000000000000000000000
I20260929 17:48:09.772774 9946 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 17:48:09.774573 10390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 17:48:09.777240 10390 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: tablet copy: Starting download of 0 data blocks...
I20260929 17:48:09.777484 10390 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: tablet copy: Starting download of 1 WAL segments...
I20260929 17:48:09.779596 10390 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 17:48:09.781767 9946 tablet_copy_service.cc:342] P 8b694f0ceec04751a088dc6d859267bb: Request end of tablet copy session 8390d94a4ac14b62a5abb1b0cf25f68c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.183.253@KRBTEST.COM'} at 127.0.0.1:60270
I20260929 17:48:09.781939 9946 tablet_copy_service.cc:433] P 8b694f0ceec04751a088dc6d859267bb: ending tablet copy session 8390d94a4ac14b62a5abb1b0cf25f68c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8390d94a4ac14b62a5abb1b0cf25f68c
I20260929 17:48:09.791314 10390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:09.793088 10499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.795914 10500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.796068 10502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:09.796346 10390 server_base.cc:1034] running on GCE node
I20260929 17:48:09.796502 10390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:09.796818 10390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:09.797967 10390 hybrid_clock.cc:648] HybridClock initialized: now 1790704089797954 us; error 29 us; skew 500 ppm
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:09.799562 10390 init.cc:377] Logged in from keytab as oryx/127.3.183.253@KRBTEST.COM (short username oryx)
I20260929 17:48:09.800094 10390 webserver.cc:460] Webserver started at http://127.3.183.253:35435/ using document root <none> and password file <none>
I20260929 17:48:09.800299 10390 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:09.800340 10390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:09.801664 10390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:09.802659 10509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.802909 10390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:09.803013 10390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "51f116b22955ee10cefa514dcdf32c0c"
server_key_iv: "1904b0cdcbbe5a1bdc6496c535673ce3"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.803167 10390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:09.816440 10390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:09.818686 10390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:09.823874 10390 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.253:38113
I20260929 17:48:09.824076 10561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.253:38113 every 8 connection(s)
I20260929 17:48:09.824862 10390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
I20260929 17:48:09.826802 10562 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:48:09.827766 10562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Bootstrap starting.
I20260929 17:48:09.838030 10562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:48:09.838362 10562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: Bootstrap complete.
I20260929 17:48:09.838989 10562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } }
I20260929 17:48:09.839126 10562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8390d94a4ac14b62a5abb1b0cf25f68c, State: Initialized, Role: LEARNER
I20260929 17:48:09.839231 10562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } }
I20260929 17:48:09.839695 10562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:09.839684 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } }
I20260929 17:48:09.840003 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: LEARNER
I20260929 17:48:09.843986 10576 catalog_manager.cc:1259] Loaded cluster ID: 4cc877435e54456b945bc782e7d97106
I20260929 17:48:09.844058 10576 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: loading cluster ID for follower catalog manager: success
I20260929 17:48:09.846210 10576 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: acquiring CA information for follower catalog manager: success
I20260929 17:48:09.847615 10576 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 17:48:09.903584 10497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:09.903947 10497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:09.904019 10497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:09.908283 10497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:09.908344 10497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:09.908368 10497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:09.908391 10497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:09.908419 10497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:09.913975 10497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.252.keytab
--principal=oryx/127.3.183.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.183.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:09.914986 10497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:09.916002 10497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:48:09.925293 10497 server_base.cc:1034] running on GCE node
W20260929 17:48:09.925246 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.925331 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:09.925242 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:09.926035 10497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:09.927065 10497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:09.928220 10497 hybrid_clock.cc:648] HybridClock initialized: now 1790704089928204 us; error 29 us; skew 500 ppm
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:09.931156 10497 init.cc:377] Logged in from keytab as oryx/127.3.183.252@KRBTEST.COM (short username oryx)
I20260929 17:48:09.931995 10497 webserver.cc:460] Webserver started at http://127.3.183.252:42555/ using document root <none> and password file <none>
I20260929 17:48:09.932503 10497 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:09.932565 10497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:09.932744 10497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:09.934847 10497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/instance:
uuid: "13a880bc74694fdd9afca5e092befe0d"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "bc4dca15f3ac34991f3bb258bf1c8250"
server_key_iv: "3722ebbcde12f96572a5b33dc5dee471"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.935410 10497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal/instance:
uuid: "13a880bc74694fdd9afca5e092befe0d"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "bc4dca15f3ac34991f3bb258bf1c8250"
server_key_iv: "3722ebbcde12f96572a5b33dc5dee471"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.939145 10497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:48:09.941892 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:09.943135 10497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:09.943255 10497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
uuid: "13a880bc74694fdd9afca5e092befe0d"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "bc4dca15f3ac34991f3bb258bf1c8250"
server_key_iv: "3722ebbcde12f96572a5b33dc5dee471"
server_key_version: "encryptionkey@0"
I20260929 17:48:09.943349 10497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:09.963450 10497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:09.966637 10497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:09.966908 10497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:09.980080 10497 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.252:44775
I20260929 17:48:09.980415 10643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.252:44775 every 8 connection(s)
I20260929 17:48:09.981181 10497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
I20260929 17:48:09.984268 10644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:09.988035 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 10497
I20260929 17:48:09.989157 10644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.252@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
Sep 29 17:48:09 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.183.252@KRBTEST.COM
I20260929 17:48:10.002902 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.991216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.3.183.254:45769)
0929 17:48:09.991346 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.991351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.991659 (+ 308us) server_negotiation.cc:407] Connection header received
0929 17:48:09.991982 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.991985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.992031 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.992105 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.993339 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.994139 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.995985 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.996304 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.996431 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.999367 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.999371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.999374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.999404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.000892 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.001422 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.001426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.001428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.001503 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.001810 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.001813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.001815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.002001 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.002083 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.002603 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.002737 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:48:10.003839 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.989542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.3.183.252:44775)
0929 17:48:09.990043 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:09.990083 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:09.990128 (+ 45us) server_negotiation.cc:407] Connection header received
0929 17:48:09.991391 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:09.991411 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:09.991761 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:09.992027 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:09.992905 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.994484 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.995751 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.996505 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.996641 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.999855 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:09.999862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:09.999877 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:09.999902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.001728 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.002138 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.002143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.002144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.002214 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.002462 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.002465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.002467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.002688 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.002857 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.003421 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.003558 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":342,"thread_start_us":116,"threads_started":1}
I20260929 17:48:10.003860 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:09.990987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60286)
0929 17:48:09.991515 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:09.991564 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:09.991758 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:09.992174 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:09.992179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:09.992769 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:09.993183 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.993206 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.994380 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.994383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.995856 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:09.995866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:09.996503 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:09.996507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:09.996668 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:09.997403 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:09.997423 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:09.999217 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.000990 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.000999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.001011 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.001332 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.001584 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.001587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.001589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.001726 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.002098 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.002105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.002377 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.002544 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.002873 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":184,"threads_started":1}
I20260929 17:48:10.005898 10644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.252@KRBTEST.COM for oryx/127.3.183.253@KRBTEST.COM
I20260929 17:48:10.014590 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.006545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.3.183.253:38113)
0929 17:48:10.006819 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.006822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.006834 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:48:10.006871 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.006873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.006910 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.006980 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.007424 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.008137 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.008854 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.009066 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.009149 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.011333 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.011336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.011338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.011363 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.013077 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.013464 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.013468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.013470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.013560 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.013798 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.013801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.013802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.013996 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.014079 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.014327 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.014438 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":123,"threads_started":1}
I20260929 17:48:10.014588 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.006415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.0.0.1:46478)
0929 17:48:10.006566 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.006580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.006667 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.007003 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.007005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.007169 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.007338 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.007345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.008236 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.008239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.008772 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.008779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.009181 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.009185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.009324 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.009846 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.009857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.011179 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.013179 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.013182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.013185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.013374 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.013640 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.013643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.013645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.013720 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.014082 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.014085 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.014179 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.014264 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.014406 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 17:48:10.015635 10644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704089, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.252@KRBTEST.COM for oryx/127.3.183.252@KRBTEST.COM
I20260929 17:48:10.023206 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.016241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.3.183.252:44775)
0929 17:48:10.016418 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.016422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.016433 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:48:10.016459 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.016461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.016495 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.016554 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.016882 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.017457 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.018168 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.018839 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.018928 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.020711 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.020715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.020717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.020739 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.021831 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.022189 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.022192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.022194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.022258 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.022469 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.022472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.022474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.022638 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.022715 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.022924 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.023061 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":118}
I20260929 17:48:10.023206 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.016173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.0.0.1:59220)
0929 17:48:10.016272 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.016285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.016362 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.016574 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.016576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.016727 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.016790 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.016795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.017589 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.017592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.018071 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.018078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.018938 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.018941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.019037 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.019406 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.019416 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.020587 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.021923 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.021926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.021928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.022089 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.022330 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.022333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.022335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.022400 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.022715 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.022718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.022796 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.022876 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.023009 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":16}
I20260929 17:48:10.026472 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Bootstrap starting.
I20260929 17:48:10.029317 10644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:10.030364 10644 log.cc:824] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:10.032404 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: No bootstrap required, opened a new log
I20260929 17:48:10.035665 10117 heartbeater.cc:503] Master 127.3.183.254:45769 was elected leader, sending a full tablet report...
I20260929 17:48:10.036901 10644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } }
I20260929 17:48:10.037117 10644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:10.037191 10644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a880bc74694fdd9afca5e092befe0d, State: Initialized, Role: FOLLOWER
I20260929 17:48:10.037600 10644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } }
I20260929 17:48:10.038153 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } } }
I20260929 17:48:10.038285 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:48:10.038844 10644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 17:48:10.044909 10661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:48:10.045003 10661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:48:10.047982 10497 master_runner.cc:428] Detected that this master 13a880bc74694fdd9afca5e092befe0d is joining an existing cluster
I20260929 17:48:10.048043 10497 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.183.252:44775
I20260929 17:48:10.057298 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9988
I20260929 17:48:10.063454 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:39299
--local_ip_for_outbound_sockets=127.3.183.193
--tserver_master_addrs=127.3.183.254:45769
--webserver_port=32975
--webserver_interface=127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab
--principal=oryx/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:48:10.089592 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.050409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.3.183.253:38113)
0929 17:48:10.050538 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.050543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.051677 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 17:48:10.051762 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.051765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.051837 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.051912 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.052925 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.053770 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055266 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.055663 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055746 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.086062 (+ 30316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.086067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.086069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.086099 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.087650 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.088153 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.088156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.088159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.088240 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.088513 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.088516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.088518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.088766 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.088835 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.089278 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.089420 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:48:10.090533 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.051207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.0.0.1:59232)
0929 17:48:10.051970 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.051982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.052110 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.052396 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.052399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.052841 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.053048 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.053056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.053958 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.053961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.054975 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.054983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.055593 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.055715 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.084488 (+ 28773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.084509 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.084972 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.089012 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.089014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.089016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.089200 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.089424 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.089426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.089428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.089498 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.090004 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.090007 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.090093 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.090180 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.090308 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":23,"thread_start_us":113,"threads_started":1}
I20260929 17:48:10.090355 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.050671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.0.0.1:46480)
0929 17:48:10.051557 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.051570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.051681 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.052029 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.052032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.052413 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.052776 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.052786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.053931 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.053934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.055125 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.055136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.055938 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.056075 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.056644 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.056662 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.085857 (+ 29195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.087799 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.087803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.087805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.088042 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.088342 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.088344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.088346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.088425 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.088905 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.088907 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.089017 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.089792 (+ 775us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.090120 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":60,"spinlock_wait_cycles":768,"thread_start_us":115,"threads_started":1}
I20260929 17:48:10.091480 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.051171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.3.183.252:44775)
0929 17:48:10.051269 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.051273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.052064 (+ 791us) server_negotiation.cc:407] Connection header received
0929 17:48:10.052216 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.052219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.052255 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.052298 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.053166 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.053846 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055066 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.055345 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055406 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.085145 (+ 29739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.085148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.085168 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.085191 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.087678 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.089283 (+ 1605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.089286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.089287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.089349 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.089574 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.089577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.089579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.089767 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.089845 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.091247 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.091352 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:48:10.092650 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.050318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60302)
0929 17:48:10.051047 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.051062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.051209 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.051534 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.051537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.052010 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.052299 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.052307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.053353 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.053355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.054658 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.054668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.055331 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.055462 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.056134 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.056157 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.084637 (+ 28480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.088688 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.088691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.088693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.088906 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.091009 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.091012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.091014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.091113 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.092170 (+ 1057us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.092173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.092270 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.092360 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.092483 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":4,"thread_start_us":187,"threads_started":1}
I20260929 17:48:10.093293 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.050415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.3.183.254:45769)
0929 17:48:10.050565 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.050569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.051132 (+ 563us) server_negotiation.cc:407] Connection header received
0929 17:48:10.051353 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.051357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.051413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.051520 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.052426 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.053233 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.054782 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.055110 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.055178 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.084883 (+ 29705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.084888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.084891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.084925 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.088555 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.090413 (+ 1858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.090417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.090418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.090513 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.091804 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.091807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.091809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.092036 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.092126 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.093027 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.093152 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704088, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.254@KRBTEST.COM for oryx/127.3.183.252@KRBTEST.COM
I20260929 17:48:10.106350 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.097127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.0.0.1:59236)
0929 17:48:10.097262 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.097277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.097355 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.097631 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.097634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.097821 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.097938 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.097945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.099028 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.099032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.099921 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.099931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.100494 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.100498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.100592 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.101099 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.101113 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.102722 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.104782 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.104785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.104787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.105061 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.105315 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.105317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.105319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.105408 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.105824 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.105827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.105925 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.106029 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.106172 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 17:48:10.107154 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.097204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.3.183.252:44775)
0929 17:48:10.097301 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.097304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.097316 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:48:10.097470 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.097473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.097523 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.097593 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.098043 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.098894 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.100058 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.100380 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.100486 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.102902 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.102906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.102910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.102957 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.104669 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.105152 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.105155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.105157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.105235 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.105486 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.105489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.105491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.105693 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.105779 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.106885 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.107010 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:48:10.108227 9926 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.183.252:44775: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "13a880bc74694fdd9afca5e092befe0d"
member_type: NON_VOTER
last_known_addr {
host: "127.3.183.252"
port: 44775
}
attrs {
promote: true
}
}
dest_uuid: "8b694f0ceec04751a088dc6d859267bb"
cas_config_opid_index: 5
I20260929 17:48:10.108793 9926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } }
I20260929 17:48:10.109802 10486 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 13a880bc74694fdd9afca5e092befe0d (127.3.183.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.110810 10618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:10.111356 10486 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.183.252:44775
I20260929 17:48:10.111510 10486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.111596 10486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.112250 10618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Refusing update from remote peer 8b694f0ceec04751a088dc6d859267bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 17:48:10.112593 10650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:10.112722 10650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } }
I20260929 17:48:10.112808 10485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:10.113570 10650 leader_election.cc:288] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769), 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253:38113)
I20260929 17:48:10.114156 10497 master.cc:537] Master@127.3.183.252:44775 shutting down...
I20260929 17:48:10.114203 10536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13a880bc74694fdd9afca5e092befe0d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" is_pre_election: true
I20260929 17:48:10.114367 10536 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 LEARNER]: Handling vote request from an unknown peer 13a880bc74694fdd9afca5e092befe0d
I20260929 17:48:10.114475 10536 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 13a880bc74694fdd9afca5e092befe0d for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 17:48:10.115326 9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13a880bc74694fdd9afca5e092befe0d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b694f0ceec04751a088dc6d859267bb" is_pre_election: true
I20260929 17:48:10.116135 10595 leader_election.cc:302] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13a880bc74694fdd9afca5e092befe0d; no voters: 8390d94a4ac14b62a5abb1b0cf25f68c, 8b694f0ceec04751a088dc6d859267bb
I20260929 17:48:10.116549 10650 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:48:10.120723 10618 raft_consensus.cc:858] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } }
W20260929 17:48:10.122656 10618 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } }
I20260929 17:48:10.124486 10618 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } }
I20260929 17:48:10.126621 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b694f0ceec04751a088dc6d859267bb. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } }
I20260929 17:48:10.126722 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 17:48:10.126801 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } }
I20260929 17:48:10.126852 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 17:48:10.126848 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.109992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.0.0.1:46496)
0929 17:48:10.122412 (+ 12420us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.122429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.122534 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.122924 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.122927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.122947 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:10.123043 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.123049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.124063 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.124066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.125315 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.125325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.126408 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.126412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.126509 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.126555 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.126640 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.126717 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12334}
I20260929 17:48:10.126902 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } }
I20260929 17:48:10.126964 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 17:48:10.127250 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.110282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.3.183.253:38113)
0929 17:48:10.117756 (+ 7474us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.117761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.122504 (+ 4743us) server_negotiation.cc:407] Connection header received
0929 17:48:10.122639 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.122643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.122705 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.122794 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:10.123158 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.123930 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.125450 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.126104 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.126178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.126318 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.127057 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.127137 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7386}
I20260929 17:48:10.127650 10536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 LEARNER]: Refusing update from remote peer 8b694f0ceec04751a088dc6d859267bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 17:48:10.128089 10485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 17:48:10.129763 10536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 13a880bc74694fdd9afca5e092befe0d (127.3.183.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.132241 10485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.132354 10485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.132967 10497 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 17:48:10.133212 10497 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 17:48:10.133285 10497 tablet_replica.cc:330] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: stopping tablet replica
W20260929 17:48:10.133481 9912 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb -> Peer 13a880bc74694fdd9afca5e092befe0d (127.3.183.252:44775): Couldn't send request to peer 13a880bc74694fdd9afca5e092befe0d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 17:48:10.134317 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b694f0ceec04751a088dc6d859267bb. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.134435 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: LEARNER
I20260929 17:48:10.134521 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: NON_VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: true } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.134582 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: LEARNER
I20260929 17:48:10.137645 10485 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: attempting to promote NON_VOTER 8390d94a4ac14b62a5abb1b0cf25f68c to VOTER
I20260929 17:48:10.138053 10485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } }
I20260929 17:48:10.139534 10536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 LEARNER]: Refusing update from remote peer 8b694f0ceec04751a088dc6d859267bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 17:48:10.139864 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.139123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.0.0.1:59240)
0929 17:48:10.139239 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.139747 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 17:48:10.140465 10681 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 17:48:10.140549 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.140337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.0.0.1:59244)
0929 17:48:10.140441 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.140489 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 17:48:10.140704 9912 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb -> Peer 13a880bc74694fdd9afca5e092befe0d (127.3.183.252:44775): Couldn't send request to peer 13a880bc74694fdd9afca5e092befe0d. Status: Network error: Client connection negotiation failed: client connection to 127.3.183.252:44775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 17:48:10.142351 10680 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.144155 10486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.144266 10486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.144488 10486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.144572 10486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.144551 10536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.146175 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.146286 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:48:10.148909 10497 master.cc:552] Master@127.3.183.252:44775 shutdown complete.
I20260929 17:48:10.148960 10497 master_runner.cc:305] Clearing existing system tablet
I20260929 17:48:10.149201 10497 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:10.149255 10497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:10.150484 10497 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 17:48:10.151351 10683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.151607 10497 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:10.151733 10497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:10.151772 10497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
uuid: "13a880bc74694fdd9afca5e092befe0d"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "bc4dca15f3ac34991f3bb258bf1c8250"
server_key_iv: "3722ebbcde12f96572a5b33dc5dee471"
server_key_version: "encryptionkey@0"
I20260929 17:48:10.153527 10497 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:10.155846 10497 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 17:48:10.155932 10497 log.cc:1196] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 17:48:10.156198 10497 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Deleting consensus metadata
I20260929 17:48:10.156355 10497 master_runner.cc:315] Copying system tablet from 127.3.183.254:45769
I20260929 17:48:10.157466 10497 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: tablet copy: Beginning tablet copy session from remote peer at address 127.3.183.254:45769
I20260929 17:48:10.166620 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.158016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60306)
0929 17:48:10.158308 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.158322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.158434 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.158694 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.158697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.158903 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.159044 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.159052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.160115 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.160118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.161036 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.161047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.161623 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.161626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.161724 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.162264 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.162278 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.162697 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.164839 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.164842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.164845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.165109 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.165496 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.165499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.165501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.165597 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.166054 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.166057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.166174 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.166279 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.166422 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
I20260929 17:48:10.166621 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.158086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.3.183.254:45769)
0929 17:48:10.158208 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.158213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.158402 (+ 189us) server_negotiation.cc:407] Connection header received
0929 17:48:10.158516 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.158519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.158571 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.158659 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.159155 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.160006 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.161163 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.161500 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.161602 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.162847 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.162851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.162853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.162881 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.164720 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.165205 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.165209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.165211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.165307 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.165684 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.165687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.165689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.165923 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.166020 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.166341 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.166459 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:48:10.167619 9946 tablet_copy_service.cc:140] P 8b694f0ceec04751a088dc6d859267bb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 13a880bc74694fdd9afca5e092befe0d ({username='oryx', principal='oryx/127.3.183.252@KRBTEST.COM'} at 127.0.0.1:60306)
I20260929 17:48:10.167719 9946 tablet_copy_service.cc:161] P 8b694f0ceec04751a088dc6d859267bb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 13a880bc74694fdd9afca5e092befe0d at {username='oryx', principal='oryx/127.3.183.252@KRBTEST.COM'} at 127.0.0.1:60306: session id = 13a880bc74694fdd9afca5e092befe0d-00000000000000000000000000000000
I20260929 17:48:10.168843 9946 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 17:48:10.170213 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 17:48:10.172569 10497 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: tablet copy: Starting download of 0 data blocks...
I20260929 17:48:10.172750 10497 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: tablet copy: Starting download of 1 WAL segments...
I20260929 17:48:10.174149 10497 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 17:48:10.176000 9946 tablet_copy_service.cc:342] P 8b694f0ceec04751a088dc6d859267bb: Request end of tablet copy session 13a880bc74694fdd9afca5e092befe0d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.183.252@KRBTEST.COM'} at 127.0.0.1:60306
I20260929 17:48:10.176079 9946 tablet_copy_service.cc:433] P 8b694f0ceec04751a088dc6d859267bb: ending tablet copy session 13a880bc74694fdd9afca5e092befe0d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 13a880bc74694fdd9afca5e092befe0d
I20260929 17:48:10.183965 10497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:10.185590 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:10.185662 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:10.186183 10497 server_base.cc:1034] running on GCE node
W20260929 17:48:10.186270 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:10.186551 10497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:10.186830 10497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:10.187944 10497 hybrid_clock.cc:648] HybridClock initialized: now 1790704090187929 us; error 32 us; skew 500 ppm
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:10.189402 10497 init.cc:377] Logged in from keytab as oryx/127.3.183.252@KRBTEST.COM (short username oryx)
I20260929 17:48:10.189817 10497 webserver.cc:460] Webserver started at http://127.3.183.252:41521/ using document root <none> and password file <none>
I20260929 17:48:10.189993 10497 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:10.190029 10497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:10.191226 10497 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:10.192188 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.192410 10497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:10.192482 10497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
uuid: "13a880bc74694fdd9afca5e092befe0d"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "bc4dca15f3ac34991f3bb258bf1c8250"
server_key_iv: "3722ebbcde12f96572a5b33dc5dee471"
server_key_version: "encryptionkey@0"
I20260929 17:48:10.192555 10497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:10.202574 10497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:10.204770 10497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:10.211181 10497 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.252:44775
I20260929 17:48:10.211650 10755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.252:44775 every 8 connection(s)
I20260929 17:48:10.211807 10497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
I20260929 17:48:10.213650 10756 sys_catalog.cc:263] Verifying existing consensus state
W20260929 17:48:10.214222 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:10.214532 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:10.214609 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 17:48:10.214690 10756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Bootstrap starting.
W20260929 17:48:10.220019 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:10.220085 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:10.220175 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:10.222395 10756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:48:10.222671 10756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Bootstrap complete.
I20260929 17:48:10.223201 10756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } }
I20260929 17:48:10.223332 10756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13a880bc74694fdd9afca5e092befe0d, State: Initialized, Role: LEARNER
I20260929 17:48:10.223433 10756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } }
I20260929 17:48:10.223749 10759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.223858 10756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:10.223912 10759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: LEARNER
I20260929 17:48:10.226610 10770 catalog_manager.cc:1259] Loaded cluster ID: 4cc877435e54456b945bc782e7d97106
I20260929 17:48:10.226665 10770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: loading cluster ID for follower catalog manager: success
I20260929 17:48:10.227355 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab
--principal=oryx/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:39299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=32975
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:10.228276 10770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: acquiring CA information for follower catalog manager: success
I20260929 17:48:10.228428 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:10.229300 10770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:48:10.229454 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:10.239974 10773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:10.239974 10772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:10.239993 10775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:10.240192 10670 server_base.cc:1034] running on GCE node
I20260929 17:48:10.240742 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:10.241410 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:10.242578 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790704090242570 us; error 45 us; skew 500 ppm
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:10.245759 10670 init.cc:377] Logged in from keytab as oryx/127.3.183.193@KRBTEST.COM (short username oryx)
I20260929 17:48:10.246591 10670 webserver.cc:460] Webserver started at http://127.3.183.193:32975/ using document root <none> and password file <none>
I20260929 17:48:10.247120 10670 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:10.247189 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:10.250316 10251 heartbeater.cc:503] Master 127.3.183.254:45769 was elected leader, sending a full tablet report...
I20260929 17:48:10.251323 10670 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:48:10.253468 10782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.254716 10670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:10.254822 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "a7b7278a49834eba9417ddb666333110"
format_stamp: "Formatted at 2026-09-29 17:48:08 on dist-test-slave-wbpt"
server_key: "3ef7832a2f47129b7159ed39016e5571"
server_key_iv: "4411a9059f4c1daa4ce365f57a372350"
server_key_version: "encryptionkey@0"
I20260929 17:48:10.255415 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:10.265218 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:10.268041 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:10.268234 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:10.268874 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:10.269717 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:10.269760 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.269800 10670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:10.269829 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.286281 10670 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:39299
I20260929 17:48:10.286334 10895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:39299 every 8 connection(s)
I20260929 17:48:10.287338 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:10.297369 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 10670
I20260929 17:48:10.297572 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 10122
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.252@KRBTEST.COM
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.253@KRBTEST.COM
I20260929 17:48:10.303705 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.289855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:41567 (local address 127.3.183.254:45769)
0929 17:48:10.289996 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.290001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.290491 (+ 490us) server_negotiation.cc:407] Connection header received
0929 17:48:10.291638 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.291642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.291698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.291808 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.293497 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.294054 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.295805 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.296119 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.296228 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.299843 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.299847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.299849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.299877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.301663 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302159 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.302163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.302164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.302248 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302526 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.302530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.302532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.302745 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.302809 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.303448 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.303551 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:48:10.303705 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.290727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:43977 (local address 127.3.183.253:38113)
0929 17:48:10.290849 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.290853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.291973 (+ 1120us) server_negotiation.cc:407] Connection header received
0929 17:48:10.292039 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.292042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.292095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.292177 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.293574 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.294342 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.295762 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.296080 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.296166 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.300336 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.300340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.300342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.300370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.301854 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302260 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.302263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.302265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.302342 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302757 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.302760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.302762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.302963 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.303033 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.303421 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.303541 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:48:10.304639 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.289245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:36371 (local address 127.3.183.252:44775)
0929 17:48:10.289527 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.289532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.291026 (+ 1494us) server_negotiation.cc:407] Connection header received
0929 17:48:10.291644 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.291647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.291702 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.291795 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.293510 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.294551 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.295539 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.295815 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.295908 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.299573 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.299577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.299579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.299606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.301447 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302038 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.302042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.302044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.302126 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302458 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.302461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.302463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.302663 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.302744 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.304261 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.304438 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260929 17:48:10.305087 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.290630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.3.183.193:43977)
0929 17:48:10.291860 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.291872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.291971 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.292214 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.292217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.292677 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.293341 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.293356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.294450 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.294453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.295641 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.295652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.296179 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.296182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.296333 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.296959 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.296975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.300212 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.301962 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.301966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.301968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.302163 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.302565 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.302571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.302654 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.303097 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.303100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.303171 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.303912 (+ 741us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.304070 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":230,"spinlock_wait_cycles":26240,"thread_start_us":75,"threads_started":1}
I20260929 17:48:10.305049 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.289645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:41567)
0929 17:48:10.290328 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.290378 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.291438 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.292051 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.292053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.292680 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.293341 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.293357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.294263 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.294265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.295641 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.295655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.296428 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.296439 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.296536 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.297089 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.297103 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.299697 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.301790 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.301794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.301796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.302041 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.302353 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.302358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.302435 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.302875 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.302878 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.303167 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.303872 (+ 705us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.304018 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":139,"spinlock_wait_cycles":40448,"thread_start_us":225,"threads_started":1}
I20260929 17:48:10.305648 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.289653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.3.183.193:36371)
0929 17:48:10.290909 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.290921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.291483 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.291991 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.292000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.292676 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.293340 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.293354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.294641 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.294644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.295428 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.295445 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.295948 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.295951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.296176 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.296828 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.296844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.299407 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.301565 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.301571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.301586 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.301927 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.302210 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.302213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.302215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.302367 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.302791 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.302797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.303159 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.303387 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.303698 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":113,"spinlock_wait_cycles":21504,"thread_start_us":169,"threads_started":1}
I20260929 17:48:10.306309 10898 heartbeater.cc:347] Connected to a master server at 127.3.183.253:38113
I20260929 17:48:10.306543 10898 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.306777 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:37975
--local_ip_for_outbound_sockets=127.3.183.194
--tserver_master_addrs=127.3.183.254:45769
--webserver_port=34253
--webserver_interface=127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab
--principal=oryx/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:48:10.306815 10896 heartbeater.cc:347] Connected to a master server at 127.3.183.252:44775
I20260929 17:48:10.306885 10896 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.307143 10898 heartbeater.cc:511] Master 127.3.183.253:38113 requested a full tablet report, sending...
I20260929 17:48:10.307156 10896 heartbeater.cc:511] Master 127.3.183.252:44775 requested a full tablet report, sending...
I20260929 17:48:10.308715 10720 ts_manager.cc:194] Registered new tserver with Master: a7b7278a49834eba9417ddb666333110 (127.3.183.193:39299)
I20260929 17:48:10.308704 10526 ts_manager.cc:194] Registered new tserver with Master: a7b7278a49834eba9417ddb666333110 (127.3.183.193:39299)
I20260929 17:48:10.309068 10897 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45769
I20260929 17:48:10.309154 10897 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.309317 10897 heartbeater.cc:511] Master 127.3.183.254:45769 requested a full tablet report, sending...
I20260929 17:48:10.310043 9926 ts_manager.cc:194] Re-registered known tserver with Master: a7b7278a49834eba9417ddb666333110 (127.3.183.193:39299)
I20260929 17:48:10.310631 9926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:41567
W20260929 17:48:10.417446 10904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:10.417689 10904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:10.417747 10904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:10.421713 10904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:10.421770 10904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:10.421845 10904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:10.426959 10904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab
--principal=oryx/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:37975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=34253
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:10.427987 10904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:10.428963 10904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:10.437994 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:10.438007 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:10.437996 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:10.438179 10904 server_base.cc:1034] running on GCE node
I20260929 17:48:10.438625 10904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:10.439191 10904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:10.440353 10904 hybrid_clock.cc:648] HybridClock initialized: now 1790704090440332 us; error 31 us; skew 500 ppm
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:10.442934 10904 init.cc:377] Logged in from keytab as oryx/127.3.183.194@KRBTEST.COM (short username oryx)
I20260929 17:48:10.443701 10904 webserver.cc:460] Webserver started at http://127.3.183.194:34253/ using document root <none> and password file <none>
I20260929 17:48:10.444216 10904 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:10.444291 10904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:10.448046 10904 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:48:10.450186 10919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.451642 10904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:48:10.451733 10904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "d7f0d93473db461695d14be89edf0640"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "203d90607ac66e11c8e1597f9b8ad369"
server_key_iv: "b8e7b7f3b3f7a4a5fe0b775fc7f649a4"
server_key_version: "encryptionkey@0"
I20260929 17:48:10.452307 10904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:10.465088 10385 heartbeater.cc:503] Master 127.3.183.254:45769 was elected leader, sending a full tablet report...
I20260929 17:48:10.479867 10904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:10.482731 10904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:10.482877 10904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:10.483397 10904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:10.484268 10904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:10.484314 10904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.484351 10904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:10.484382 10904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.492295 10904 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:37975
I20260929 17:48:10.492324 11032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:37975 every 8 connection(s)
I20260929 17:48:10.493280 10904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:10.503268 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 10904
I20260929 17:48:10.503468 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 10256
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.253@KRBTEST.COM
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.252@KRBTEST.COM
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:10.509883 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:44667
--local_ip_for_outbound_sockets=127.3.183.195
--tserver_master_addrs=127.3.183.254:45769
--webserver_port=40417
--webserver_interface=127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab
--principal=oryx/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:48:10.510072 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.495046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:50375 (local address 127.3.183.253:38113)
0929 17:48:10.495167 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.495171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.496494 (+ 1323us) server_negotiation.cc:407] Connection header received
0929 17:48:10.498048 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.498051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.498104 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.498177 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.499438 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.500256 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.501535 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.501802 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.501900 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.505976 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.505980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.505983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.506012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.507597 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.508207 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.508211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.508213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.508411 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.508839 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.508843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.508845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.509079 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.509174 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.509783 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.509904 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:48:10.510762 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.496990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:36907 (local address 127.3.183.252:44775)
0929 17:48:10.497114 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.497118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.497305 (+ 187us) server_negotiation.cc:407] Connection header received
0929 17:48:10.497786 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.497789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.497838 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.497890 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.499409 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.500256 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.501691 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.502001 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.502082 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.506469 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.506473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.506476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.506505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.508476 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.509514 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.509517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.509519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.509607 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.509900 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.509903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.509905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.510117 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.510183 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.510496 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.510621 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:48:10.511644 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.495046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:41603 (local address 127.3.183.254:45769)
0929 17:48:10.495167 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.495171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.496493 (+ 1322us) server_negotiation.cc:407] Connection header received
0929 17:48:10.497780 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.497784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.497838 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.497911 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.499430 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.500006 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.502177 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.502389 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.502483 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.506814 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.506819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.506821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.506852 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.509104 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.509831 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.509834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.509836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.509916 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.510600 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.510604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.510605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.510801 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.510873 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.511404 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.511513 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:48:10.511837 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.496858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.3.183.194:36907)
0929 17:48:10.497203 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.497214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.497555 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.498087 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.498096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.498604 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.499266 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.499280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.500641 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.500644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.501584 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.501594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.502122 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.502126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.502245 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.502890 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.502902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.506322 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.509207 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.509210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.509212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.509426 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.509715 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.509718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.509720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.509803 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.510254 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.510257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.510339 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.510889 (+ 550us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.511039 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":190,"spinlock_wait_cycles":3456,"thread_start_us":118,"threads_started":1}
I20260929 17:48:10.511869 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.495588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.3.183.194:50375)
0929 17:48:10.496324 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.496383 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.497555 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.498210 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.498212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.498628 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.499260 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.499280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.500361 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.500364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.501374 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.501383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.501936 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.501939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.502138 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.502861 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.502882 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.505790 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.507705 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.507712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.507729 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.508087 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.508530 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.508533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.508535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.508723 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.509189 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.509196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.509493 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.510285 (+ 792us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.510632 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":154,"spinlock_wait_cycles":55424,"thread_start_us":311,"threads_started":1}
I20260929 17:48:10.512524 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.495588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:41603)
0929 17:48:10.496324 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.496383 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.497557 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.498087 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.498094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.498640 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.499260 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.499280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.500207 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.500209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.501584 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.501594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.502600 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.502604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.502701 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.503254 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.503269 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.506638 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.509527 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.509529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.509531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.509732 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.510411 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.510414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.510415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.510498 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.511149 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.511152 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.511234 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.512246 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.512376 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":127,"spinlock_wait_cycles":19456,"thread_start_us":373,"threads_started":1}
I20260929 17:48:10.513270 11035 heartbeater.cc:347] Connected to a master server at 127.3.183.253:38113
I20260929 17:48:10.513342 11033 heartbeater.cc:347] Connected to a master server at 127.3.183.252:44775
I20260929 17:48:10.513478 11035 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.513479 11033 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.513844 11034 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45769
I20260929 17:48:10.513908 11034 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.514000 11035 heartbeater.cc:511] Master 127.3.183.253:38113 requested a full tablet report, sending...
I20260929 17:48:10.514000 11033 heartbeater.cc:511] Master 127.3.183.252:44775 requested a full tablet report, sending...
I20260929 17:48:10.514088 11034 heartbeater.cc:511] Master 127.3.183.254:45769 requested a full tablet report, sending...
I20260929 17:48:10.515125 9926 ts_manager.cc:194] Re-registered known tserver with Master: d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975)
I20260929 17:48:10.515211 10720 ts_manager.cc:194] Registered new tserver with Master: d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975)
I20260929 17:48:10.515266 10526 ts_manager.cc:194] Registered new tserver with Master: d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975)
I20260929 17:48:10.515882 9926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:41603
W20260929 17:48:10.621409 11040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:10.621654 11040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:10.621697 11040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:10.625613 11040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:10.625674 11040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:10.625756 11040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:10.631009 11040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab
--principal=oryx/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:44667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=40417
--tserver_master_addrs=127.3.183.254:45769,127.3.183.253:38113,127.3.183.252:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:10.632097 11040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:10.633059 11040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:10.642741 11046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:10.642776 11045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:10.642822 11048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:10.643179 11040 server_base.cc:1034] running on GCE node
I20260929 17:48:10.643522 11040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:10.644129 11040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:10.645277 11040 hybrid_clock.cc:648] HybridClock initialized: now 1790704090645258 us; error 27 us; skew 500 ppm
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:10.647931 11040 init.cc:377] Logged in from keytab as oryx/127.3.183.195@KRBTEST.COM (short username oryx)
I20260929 17:48:10.648705 11040 webserver.cc:460] Webserver started at http://127.3.183.195:40417/ using document root <none> and password file <none>
I20260929 17:48:10.649212 11040 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:10.649289 11040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:10.652889 11040 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:10.654817 11055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.656136 11040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:10.656256 11040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "a32093fe7d284f5bb9717e285bf918da"
format_stamp: "Formatted at 2026-09-29 17:48:09 on dist-test-slave-wbpt"
server_key: "7615f18dc6552e38f42e5e399889e1a0"
server_key_iv: "1dda1113f51039b4c7a860c959e2e6e7"
server_key_version: "encryptionkey@0"
I20260929 17:48:10.656787 11040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:10.673929 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.670356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.3.183.252:44775)
0929 17:48:10.670481 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.670485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.670558 (+ 73us) server_negotiation.cc:407] Connection header received
0929 17:48:10.670676 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.670679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.670739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.670840 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:10.671028 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.671818 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.672858 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.673461 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.673548 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.673701 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.673737 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.673810 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:48:10.673944 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.670269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.0.0.1:59248)
0929 17:48:10.670482 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.670495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.670595 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.670847 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.670849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.670861 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:10.670950 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.670955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.671961 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.671964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.672747 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.672757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.673570 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.673573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.673647 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.673672 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.673728 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.673802 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260929 17:48:10.674551 10730 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 17:48:10.674813 10759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b694f0ceec04751a088dc6d859267bb. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.674964 10759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: LEARNER
I20260929 17:48:10.679430 11040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:10.682015 11040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:10.682163 11040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:10.682685 11040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:10.683470 11040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:10.683513 11040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.683550 11040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:10.683571 11040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:10.693372 11040 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:44667
I20260929 17:48:10.693406 11169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:44667 every 8 connection(s)
I20260929 17:48:10.694381 11040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:10.697723 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 11040
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.253@KRBTEST.COM
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
Sep 29 17:48:10 dist-test-slave-wbpt krb5kdc[9881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704090, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.252@KRBTEST.COM
I20260929 17:48:10.709780 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.699667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.3.183.253:38113)
0929 17:48:10.700186 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.700190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.700392 (+ 202us) server_negotiation.cc:407] Connection header received
0929 17:48:10.700564 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.700567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.700615 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.700689 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.701629 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.702388 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.703480 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.703864 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.703933 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.705679 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.705682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.705685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.705710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.707959 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.708419 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.708423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.708424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.708507 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.708841 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.708845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.708847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.709061 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.709159 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.709469 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.709588 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":72,"threads_started":1}
I20260929 17:48:10.710412 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.699718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.3.183.254:45769)
0929 17:48:10.700046 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.700050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.701302 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 17:48:10.701473 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.701476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.701524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.701581 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.702439 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.703259 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.704776 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.705174 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.705253 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.706796 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.706800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.706803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.706829 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.708525 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.708979 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.708983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.708985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.709063 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.709361 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.709364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.709366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.709565 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.709658 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.710051 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.710259 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":261,"thread_start_us":79,"threads_started":1}
I20260929 17:48:10.712560 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.697740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:50551 (local address 127.3.183.253:38113)
0929 17:48:10.697857 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.697862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.698122 (+ 260us) server_negotiation.cc:407] Connection header received
0929 17:48:10.698430 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.698434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.698484 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.698534 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.700556 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.701382 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.703174 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.703546 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.703621 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.708853 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.708857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.708859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.708885 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.710317 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.710901 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.710904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.710906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.710979 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.711360 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.711363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.711365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.711574 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.711637 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.712311 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.712416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:48:10.713016 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.696130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:59379 (local address 127.3.183.254:45769)
0929 17:48:10.696260 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.696264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.697234 (+ 970us) server_negotiation.cc:407] Connection header received
0929 17:48:10.698513 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.698516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.698573 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.698646 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.701649 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.702420 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.704579 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.704903 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.705016 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.708584 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.708588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.708590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.708615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.710438 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.711499 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.711502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.711504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.711574 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.711888 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.711891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.711893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.712087 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.712171 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.712781 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.712874 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260929 17:48:10.714570 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.697637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.3.183.195:50551)
0929 17:48:10.698011 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.698023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.698207 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.698844 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.698852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.699433 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.700336 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.700358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.701568 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.701572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.703038 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.703049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.703894 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.703899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.704160 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.705732 (+ 1572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.705751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.708706 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.710443 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.710449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.710466 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.710795 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.711085 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.711088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.711090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.711258 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.711707 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.711714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.712048 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.712691 (+ 643us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.712873 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":179,"spinlock_wait_cycles":2816,"thread_start_us":77,"threads_started":1}
I20260929 17:48:10.714785 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.696616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:59379)
0929 17:48:10.697075 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.697133 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.698207 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.698843 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.698852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.699433 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.701507 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.701517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.702563 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.702567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.703436 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.703446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.705739 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.705743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.705929 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.706461 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.706477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.708352 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.711190 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.711192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.711194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.711403 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.711670 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.711672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.711674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.711751 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.712210 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.712213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.712291 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.712386 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.713258 (+ 872us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1713,"spinlock_wait_cycles":25984,"thread_start_us":109,"threads_started":1}
I20260929 17:48:10.715582 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.696616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.3.183.195:55555)
0929 17:48:10.697074 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.697126 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.698207 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.698844 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.698852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.699549 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:10.700369 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.700377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.703559 (+ 3182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.703562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.704346 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.704356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.706143 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.706146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.706239 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.707044 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:10.707059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:10.711089 (+ 4030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:10.713213 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.713216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.713217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.713430 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.713741 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:10.713744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:10.713746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:10.713829 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:10.714245 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:10.714248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:10.714328 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.715294 (+ 966us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.715420 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":358,"spinlock_wait_cycles":43520,"thread_start_us":119,"threads_started":1}
I20260929 17:48:10.715737 11171 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45769
I20260929 17:48:10.715977 11171 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.716361 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.696148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:55555 (local address 127.3.183.252:44775)
0929 17:48:10.696356 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.696360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.697230 (+ 870us) server_negotiation.cc:407] Connection header received
0929 17:48:10.698430 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.698434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.698483 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.698534 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.700530 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.701376 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.705128 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.705485 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.705620 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.711201 (+ 5581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.711205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.711208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.711235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.712972 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.713537 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.713541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.713543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.713638 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.713925 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.713928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.713930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.714138 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.714232 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.716102 (+ 1870us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.716221 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":133}
I20260929 17:48:10.716517 11171 heartbeater.cc:511] Master 127.3.183.254:45769 requested a full tablet report, sending...
I20260929 17:48:10.717022 11172 heartbeater.cc:347] Connected to a master server at 127.3.183.253:38113
I20260929 17:48:10.717101 11172 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.717108 11170 heartbeater.cc:347] Connected to a master server at 127.3.183.252:44775
I20260929 17:48:10.717247 11170 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:10.717315 11172 heartbeater.cc:511] Master 127.3.183.253:38113 requested a full tablet report, sending...
I20260929 17:48:10.717394 11170 heartbeater.cc:511] Master 127.3.183.252:44775 requested a full tablet report, sending...
I20260929 17:48:10.717537 9925 ts_manager.cc:194] Re-registered known tserver with Master: a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667)
I20260929 17:48:10.718142 9925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:59379
I20260929 17:48:10.718240 10526 ts_manager.cc:194] Registered new tserver with Master: a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667)
I20260929 17:48:10.718773 10720 ts_manager.cc:194] Registered new tserver with Master: a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667)
I20260929 17:48:10.714236 9926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:10.721001 9926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:10.721616 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.700534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.3.183.252:44775)
0929 17:48:10.701016 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.701025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.704574 (+ 3549us) server_negotiation.cc:407] Connection header received
0929 17:48:10.704683 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.704686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.704738 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.704815 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:10.705573 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.714411 (+ 8838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.715529 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.716079 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.716163 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.718356 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:10.718360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:10.718362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:10.718386 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:10.719965 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.720381 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.720384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.720385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.720457 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:10.720763 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:10.720765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:10.720766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:10.720919 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:10.721007 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.721283 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.721403 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1998,"server-negotiator.queue_time_us":414,"thread_start_us":79,"threads_started":1}
I20260929 17:48:10.723667 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: NON_VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: true } } }
I20260929 17:48:10.723824 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.726725 11188 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: attempting to promote NON_VOTER 13a880bc74694fdd9afca5e092befe0d to VOTER
I20260929 17:48:10.727012 11188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:10.728753 10729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 LEARNER]: Refusing update from remote peer 8b694f0ceec04751a088dc6d859267bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 17:48:10.729084 11190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 17:48:10.729951 10536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Refusing update from remote peer 8b694f0ceec04751a088dc6d859267bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 17:48:10.730274 11190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 17:48:10.730650 11190 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 13a880bc74694fdd9afca5e092befe0d (127.3.183.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.731215 10729 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 13a880bc74694fdd9afca5e092befe0d (127.3.183.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.732270 11190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.732368 11190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.732620 11190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.732702 11189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.732810 11189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.732707 11190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:10.733212 10759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.733309 10759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:48:10.733321 10536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 13a880bc74694fdd9afca5e092befe0d (127.3.183.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.737126 11198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b694f0ceec04751a088dc6d859267bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:10.737305 11198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:48:10.741303 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.734445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:37975 (local address 127.0.0.1:58570)
0929 17:48:10.734588 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.734602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.734677 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.736134 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.736137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.736158 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:10.736258 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.736265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.737670 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.737673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.738777 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.738787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.740924 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.740928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.741030 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.741071 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.741114 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.741183 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 17:48:10.741874 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.735494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39299 (local address 127.0.0.1:37490)
0929 17:48:10.736805 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.736818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.736923 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.737575 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.737578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.737592 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:10.737662 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.737666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.738849 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.738888 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.740020 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.740030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.741264 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.741267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.741574 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.741600 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.741679 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.741751 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":118,"threads_started":1}
I20260929 17:48:10.742093 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.734835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.3.183.194:37975)
0929 17:48:10.735554 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.735589 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.735615 (+ 26us) server_negotiation.cc:407] Connection header received
0929 17:48:10.735698 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.735703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.735961 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.736157 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:10.736351 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.737536 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.739840 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.740798 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.741564 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.741708 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.741755 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.741843 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":149,"threads_started":1}
I20260929 17:48:10.742524 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.736253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.3.183.193:39299)
0929 17:48:10.736567 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.736605 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.736903 (+ 298us) server_negotiation.cc:407] Connection header received
0929 17:48:10.737050 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.737055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.737274 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.737455 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:10.737761 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.738734 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.740148 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.740965 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.741062 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.741204 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.742277 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.742376 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":138,"threads_started":1}
I20260929 17:48:10.743167 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.734953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:44667 (local address 127.0.0.1:42172)
0929 17:48:10.735166 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.735178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.735258 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.738068 (+ 2810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.738072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.738090 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:10.738217 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.738223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.739456 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.739460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.740682 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.740692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.742792 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.742795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.742881 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.742903 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.742971 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.743042 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260929 17:48:10.743381 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.735104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.3.183.195:44667)
0929 17:48:10.737583 (+ 2479us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.737618 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.737641 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:48:10.737709 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.737714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.737893 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.738086 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:10.738323 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.739361 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.740811 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.742499 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.742599 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.742732 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.743008 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.743105 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2376,"thread_start_us":104,"threads_started":1}
I20260929 17:48:10.746038 10967 tablet_service.cc:1467] Processing CreateTablet for tablet 958b7a6662df463381b89e83b69470ba (DEFAULT_TABLE table=test-table [id=b0646f58f992459e8a21a70c933afb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:10.745991 10830 tablet_service.cc:1467] Processing CreateTablet for tablet 958b7a6662df463381b89e83b69470ba (DEFAULT_TABLE table=test-table [id=b0646f58f992459e8a21a70c933afb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:10.747108 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958b7a6662df463381b89e83b69470ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:10.747109 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958b7a6662df463381b89e83b69470ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:10.754253 11104 tablet_service.cc:1467] Processing CreateTablet for tablet 958b7a6662df463381b89e83b69470ba (DEFAULT_TABLE table=test-table [id=b0646f58f992459e8a21a70c933afb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:10.755456 11104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958b7a6662df463381b89e83b69470ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:10.756537 11204 tablet_bootstrap.cc:492] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Bootstrap starting.
I20260929 17:48:10.758327 11205 tablet_bootstrap.cc:492] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Bootstrap starting.
I20260929 17:48:10.759620 11204 tablet_bootstrap.cc:654] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:10.760988 11204 log.cc:824] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:10.761124 11205 tablet_bootstrap.cc:654] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:10.762344 11205 log.cc:824] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:10.763208 11204 tablet_bootstrap.cc:492] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: No bootstrap required, opened a new log
I20260929 17:48:10.763399 11204 ts_tablet_manager.cc:1397] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:48:10.764178 11208 tablet_bootstrap.cc:492] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Bootstrap starting.
I20260929 17:48:10.764492 11205 tablet_bootstrap.cc:492] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: No bootstrap required, opened a new log
I20260929 17:48:10.764678 11205 ts_tablet_manager.cc:1397] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260929 17:48:10.766845 11208 tablet_bootstrap.cc:654] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:10.767971 11208 log.cc:824] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:10.769791 11208 tablet_bootstrap.cc:492] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: No bootstrap required, opened a new log
I20260929 17:48:10.769963 11208 ts_tablet_manager.cc:1397] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260929 17:48:10.769873 11204 raft_consensus.cc:348] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.770129 11204 raft_consensus.cc:374] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:10.770188 11204 raft_consensus.cc:729] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b7278a49834eba9417ddb666333110, State: Initialized, Role: FOLLOWER
I20260929 17:48:10.770048 11205 raft_consensus.cc:348] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.770246 11205 raft_consensus.cc:374] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:10.770284 11205 raft_consensus.cc:729] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f0d93473db461695d14be89edf0640, State: Initialized, Role: FOLLOWER
I20260929 17:48:10.770663 11204 consensus_queue.cc:260] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.770663 11205 consensus_queue.cc:260] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.771670 11204 ts_tablet_manager.cc:1428] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20260929 17:48:10.771662 11205 ts_tablet_manager.cc:1428] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:48:10.772109 11034 heartbeater.cc:503] Master 127.3.183.254:45769 was elected leader, sending a full tablet report...
I20260929 17:48:10.773662 10897 heartbeater.cc:503] Master 127.3.183.254:45769 was elected leader, sending a full tablet report...
I20260929 17:48:10.775662 11208 raft_consensus.cc:348] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.776002 11208 raft_consensus.cc:374] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:10.776062 11208 raft_consensus.cc:729] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32093fe7d284f5bb9717e285bf918da, State: Initialized, Role: FOLLOWER
I20260929 17:48:10.776495 11208 consensus_queue.cc:260] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.777617 11208 ts_tablet_manager.cc:1428] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260929 17:48:10.777963 11171 heartbeater.cc:503] Master 127.3.183.254:45769 was elected leader, sending a full tablet report...
W20260929 17:48:10.789525 10899 tablet.cc:2369] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:48:10.945775 11173 tablet.cc:2369] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:10.961516 11210 raft_consensus.cc:482] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:10.961759 11210 raft_consensus.cc:504] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.962960 11210 leader_election.cc:288] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975), a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667)
I20260929 17:48:10.967929 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.963282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:53941 (local address 127.3.183.195:44667)
0929 17:48:10.963412 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.963417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.963658 (+ 241us) server_negotiation.cc:407] Connection header received
0929 17:48:10.963818 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.963821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.963871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.963958 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:10.964308 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.965160 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.966742 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.967412 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.967560 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.967599 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.967714 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.967799 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:48:10.967929 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.963282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:56761 (local address 127.3.183.194:37975)
0929 17:48:10.963412 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:10.963417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:10.963659 (+ 242us) server_negotiation.cc:407] Connection header received
0929 17:48:10.963818 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:10.963821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:10.963871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:10.963973 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:10.964308 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.965127 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.966742 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.967354 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.967478 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.967534 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:10.967715 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 17:48:10.967799 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:48:10.967965 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.963190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:37975 (local address 127.3.183.193:56761)
0929 17:48:10.963584 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.963595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.963710 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.963984 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.963987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.964011 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:10.964209 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.964214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.965231 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.965235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.966601 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.966613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.967469 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.967473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.967572 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.967598 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.967695 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.967808 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":129,"spinlock_wait_cycles":9344,"thread_start_us":63,"threads_started":1}
I20260929 17:48:10.967965 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:10.963190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:44667 (local address 127.3.183.193:53941)
0929 17:48:10.963571 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:10.963586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:10.963710 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:10.963984 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:10.963987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:10.964012 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:10.964204 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.964212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.965250 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.965253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.966608 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:10.966614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:10.967547 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:10.967550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:10.967618 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:10.967635 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:48:10.967696 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:48:10.967797 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":154,"spinlock_wait_cycles":17792,"thread_start_us":67,"threads_started":1}
I20260929 17:48:10.968721 11124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958b7a6662df463381b89e83b69470ba" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32093fe7d284f5bb9717e285bf918da" is_pre_election: true
I20260929 17:48:10.968858 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958b7a6662df463381b89e83b69470ba" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f0d93473db461695d14be89edf0640" is_pre_election: true
I20260929 17:48:10.969061 11124 raft_consensus.cc:2454] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 0.
I20260929 17:48:10.969143 10987 raft_consensus.cc:2454] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 0.
I20260929 17:48:10.969528 10785 leader_election.cc:302] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a32093fe7d284f5bb9717e285bf918da, a7b7278a49834eba9417ddb666333110; no voters:
I20260929 17:48:10.969759 11210 raft_consensus.cc:2781] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:10.969839 11210 raft_consensus.cc:482] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:10.969890 11210 raft_consensus.cc:3037] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:10.971036 11210 raft_consensus.cc:504] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.971360 11210 leader_election.cc:288] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 election: Requested vote from peers d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975), a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667)
I20260929 17:48:10.971678 11124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958b7a6662df463381b89e83b69470ba" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32093fe7d284f5bb9717e285bf918da"
I20260929 17:48:10.971695 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958b7a6662df463381b89e83b69470ba" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f0d93473db461695d14be89edf0640"
I20260929 17:48:10.971822 11124 raft_consensus.cc:3037] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:10.971827 10987 raft_consensus.cc:3037] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:10.973337 11124 raft_consensus.cc:2454] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 1.
I20260929 17:48:10.973596 10785 leader_election.cc:302] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a32093fe7d284f5bb9717e285bf918da, a7b7278a49834eba9417ddb666333110; no voters:
I20260929 17:48:10.973778 11210 raft_consensus.cc:2781] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:10.973801 10987 raft_consensus.cc:2454] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 1.
I20260929 17:48:10.973973 11210 raft_consensus.cc:686] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 1 LEADER]: Becoming Leader. State: Replica: a7b7278a49834eba9417ddb666333110, State: Running, Role: LEADER
I20260929 17:48:10.974290 11210 consensus_queue.cc:237] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } }
I20260929 17:48:10.977133 9926 catalog_manager.cc:5495] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7b7278a49834eba9417ddb666333110 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "a7b7278a49834eba9417ddb666333110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } health_report { overall_health: HEALTHY } } }
W20260929 17:48:10.998260 11036 tablet.cc:2369] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:11.038913 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:11.035251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.3.183.193:39299)
0929 17:48:11.035383 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:11.035388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:11.035401 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:11.035479 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:11.035482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:11.035535 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:11.035623 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:11.036034 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.036582 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.037436 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.037655 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.037751 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:11.038063 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:11.038496 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:11.038610 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:11.038693 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 17:48:11.038756 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:48:11.045023 11124 raft_consensus.cc:1261] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Refusing update from remote peer a7b7278a49834eba9417ddb666333110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:11.045051 10987 raft_consensus.cc:1261] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Refusing update from remote peer a7b7278a49834eba9417ddb666333110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:11.045694 11210 consensus_queue.cc:1035] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:11.045872 11215 consensus_queue.cc:1035] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:11.053846 11219 mvcc.cc:205] Tried to move back new op lower bound from 7334723956913496064 to 7334723956633067520. Current Snapshot: MvccSnapshot[applied={T|T < 7334723956913496064}]
I20260929 17:48:11.057271 11220 mvcc.cc:205] Tried to move back new op lower bound from 7334723956913496064 to 7334723956633067520. Current Snapshot: MvccSnapshot[applied={T|T < 7334723956913496064}]
I20260929 17:48:11.063263 9925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:11.063436 9925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:11.067524 9925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b694f0ceec04751a088dc6d859267bb: Sending DeleteTablet for 3 replicas of tablet 958b7a6662df463381b89e83b69470ba
I20260929 17:48:11.068229 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet 958b7a6662df463381b89e83b69470ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:11 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:58570
I20260929 17:48:11.068702 11227 tablet_replica.cc:330] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: stopping tablet replica
I20260929 17:48:11.068996 11227 raft_consensus.cc:2229] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:11.069267 11227 raft_consensus.cc:2258] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:11.070223 11104 tablet_service.cc:1514] Processing DeleteTablet for tablet 958b7a6662df463381b89e83b69470ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:11 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42172
I20260929 17:48:11.070638 10830 tablet_service.cc:1514] Processing DeleteTablet for tablet 958b7a6662df463381b89e83b69470ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:11 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:37490
I20260929 17:48:11.070855 11228 tablet_replica.cc:330] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: stopping tablet replica
I20260929 17:48:11.071157 11228 raft_consensus.cc:2229] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:11.071449 11228 raft_consensus.cc:2258] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:11.072737 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 9894
I20260929 17:48:11.074195 11228 ts_tablet_manager.cc:1905] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:11.076596 11227 ts_tablet_manager.cc:1905] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:11.079659 11227 ts_tablet_manager.cc:1918] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:11.079762 11227 log.cc:1196] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/958b7a6662df463381b89e83b69470ba
I20260929 17:48:11.080108 11227 ts_tablet_manager.cc:1939] T 958b7a6662df463381b89e83b69470ba P d7f0d93473db461695d14be89edf0640: Deleting consensus metadata
W20260929 17:48:11.081346 10921 connection.cc:535] server connection from 127.0.0.1:58570 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:48:11.082749 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:11.071363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.3.183.253:38113)
0929 17:48:11.071469 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:11.071474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:11.071994 (+ 520us) server_negotiation.cc:407] Connection header received
0929 17:48:11.072094 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:11.072097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:11.072138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:11.072202 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:11.074362 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.075136 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.076526 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.076870 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.076941 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:11.079090 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:11.079094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:11.079096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:11.079121 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:11.080865 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.081344 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.081348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.081350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.081432 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.081810 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.081813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.081815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.082033 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:11.082131 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:11.082472 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 17:48:11.082583 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:48:11.083556 11228 ts_tablet_manager.cc:1918] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:11.083662 11228 log.cc:1196] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/958b7a6662df463381b89e83b69470ba
I20260929 17:48:11.083989 11229 tablet_replica.cc:330] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: stopping tablet replica
I20260929 17:48:11.084306 11229 raft_consensus.cc:2229] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:48:11.084679 11229 raft_consensus.cc:2258] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:11.085125 11228 ts_tablet_manager.cc:1939] T 958b7a6662df463381b89e83b69470ba P a32093fe7d284f5bb9717e285bf918da: Deleting consensus metadata
W20260929 17:48:11.086229 11056 connection.cc:441] server connection from 127.0.0.1:42172 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42172 (request call id 1) could send its response
I20260929 17:48:11.087841 11229 ts_tablet_manager.cc:1905] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:11.090924 11229 ts_tablet_manager.cc:1918] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:11.091024 11229 log.cc:1196] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/958b7a6662df463381b89e83b69470ba
I20260929 17:48:11.091339 11229 ts_tablet_manager.cc:1939] T 958b7a6662df463381b89e83b69470ba P a7b7278a49834eba9417ddb666333110: Deleting consensus metadata
W20260929 17:48:11.092517 10785 connection.cc:441] server connection from 127.0.0.1:37490 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37490 (request call id 1) could send its response
I20260929 17:48:11.094002 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:11.071684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.3.183.252:44775)
0929 17:48:11.071802 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:11.071808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:11.084491 (+ 12683us) server_negotiation.cc:407] Connection header received
0929 17:48:11.084542 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:11.084545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:11.084599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:11.084672 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:11.085506 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.086257 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.087299 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.087627 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.087707 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:11.089931 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:11.089935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:11.089938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:11.089966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:11.091692 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.092479 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.092482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.092484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.092595 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.092879 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.092882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.092884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.093130 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:11.093244 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:11.093703 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 17:48:11.093820 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 17:48:11.096534 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:11.084912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.3.183.253:38113)
0929 17:48:11.085014 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:11.085018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:11.087177 (+ 2159us) server_negotiation.cc:407] Connection header received
0929 17:48:11.087329 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:11.087332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:11.087371 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:11.087414 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:11.088955 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.089747 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.090870 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.091210 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.091287 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:11.093124 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:11.093128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:11.093130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:11.093152 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:11.095027 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.095413 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.095416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.095418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.095492 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.095744 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.095747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.095749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.095946 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:11.096051 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:11.096264 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 17:48:11.096365 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:48:11.101150 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:11.086145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.3.183.252:44775)
0929 17:48:11.086243 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:11.086246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:11.092952 (+ 6706us) server_negotiation.cc:407] Connection header received
0929 17:48:11.093145 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:11.093148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:11.093198 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:11.093253 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:11.094106 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.094864 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.096475 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:11.096807 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:11.096875 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:11.098508 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:11.098511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:11.098514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:11.098538 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:11.099745 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.100205 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.100207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.100209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.100275 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:11.100505 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:11.100507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:11.100508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:11.100660 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:11.100750 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:11.100947 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 17:48:11.101036 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:48:12.059870 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.059285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:57081)
0929 17:48:12.059589 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.059724 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
W20260929 17:48:12.060297 11171 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:45769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.061035 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.060512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:43575)
0929 17:48:12.060874 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.060958 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":103,"threads_started":1}
I20260929 17:48:12.061168 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.060692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:42015)
0929 17:48:12.060980 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.061069 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":64,"threads_started":1}
W20260929 17:48:12.061318 11034 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:45769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
W20260929 17:48:12.061400 10897 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:45769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.568861 11241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b694f0ceec04751a088dc6d859267bb)
I20260929 17:48:12.569084 11241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:12.569911 11242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b694f0ceec04751a088dc6d859267bb)
I20260929 17:48:12.570052 11242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:12.570178 11241 leader_election.cc:288] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769), 13a880bc74694fdd9afca5e092befe0d (127.3.183.252:44775)
I20260929 17:48:12.570611 11242 leader_election.cc:288] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769), 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253:38113)
I20260929 17:48:12.571018 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.570374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60748)
0929 17:48:12.570830 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.570914 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":60,"threads_started":1}
W20260929 17:48:12.571215 10513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.571424 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.570730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60758)
0929 17:48:12.571260 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.571332 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":147,"thread_start_us":53,"threads_started":1}
W20260929 17:48:12.571600 10704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.572199 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.571955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60766)
0929 17:48:12.572084 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.572132 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 17:48:12.572362 10513 leader_election.cc:334] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769): Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.572491 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.572267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60770)
0929 17:48:12.572389 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.572433 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 17:48:12.572590 10704 leader_election.cc:334] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769): Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.574831 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.570474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:44775 (local address 127.0.0.1:59286)
0929 17:48:12.570885 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.570897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:12.571045 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:12.571846 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:12.571849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:12.571865 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:12.571968 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:12.571976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.572925 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.572928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:12.573794 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:12.573803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.574441 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.574445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:12.574532 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:12.574571 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 17:48:12.574631 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:48:12.574705 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260929 17:48:12.574829 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.570612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.3.183.252:44775)
0929 17:48:12.570829 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:12.570834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:12.571154 (+ 320us) server_negotiation.cc:407] Connection header received
0929 17:48:12.571629 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:12.571632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:12.571674 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:12.571734 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:12.572081 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.572816 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.573908 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.574331 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.574421 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:12.574481 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:12.574655 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:48:12.574732 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260929 17:48:12.574990 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.570792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:38113 (local address 127.0.0.1:46524)
0929 17:48:12.571216 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.571229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:12.571314 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:12.571684 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:12.571687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:12.571703 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:12.571859 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:12.571865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.572976 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.572979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:12.573904 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:12.573914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.574600 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.574603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:12.574692 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:12.574710 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 17:48:12.574772 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:48:12.574850 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"spinlock_wait_cycles":896,"thread_start_us":58,"threads_started":1}
I20260929 17:48:12.574975 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.571006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.3.183.253:38113)
0929 17:48:12.571469 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:12.571473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:12.571487 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:48:12.571531 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:12.571534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:12.571577 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:12.571952 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:12.572020 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.572851 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.574059 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:12.574450 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:12.574589 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:12.574645 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:12.574794 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 17:48:12.574869 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":390,"thread_start_us":53,"threads_started":1}
I20260929 17:48:12.575469 10536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13a880bc74694fdd9afca5e092befe0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" is_pre_election: true
I20260929 17:48:12.575599 10536 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a880bc74694fdd9afca5e092befe0d in term 1.
I20260929 17:48:12.575701 10729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13a880bc74694fdd9afca5e092befe0d" is_pre_election: true
I20260929 17:48:12.575899 10703 leader_election.cc:302] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13a880bc74694fdd9afca5e092befe0d, 8390d94a4ac14b62a5abb1b0cf25f68c; no voters: 8b694f0ceec04751a088dc6d859267bb
I20260929 17:48:12.575910 10729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8390d94a4ac14b62a5abb1b0cf25f68c in term 1.
I20260929 17:48:12.576085 11242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 17:48:12.576144 11242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b694f0ceec04751a088dc6d859267bb)
I20260929 17:48:12.576192 11242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:48:12.576282 10512 leader_election.cc:302] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13a880bc74694fdd9afca5e092befe0d, 8390d94a4ac14b62a5abb1b0cf25f68c; no voters: 8b694f0ceec04751a088dc6d859267bb
I20260929 17:48:12.576475 11241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 17:48:12.576530 11241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b694f0ceec04751a088dc6d859267bb)
I20260929 17:48:12.576568 11241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:48:12.577253 11242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:12.577571 11242 leader_election.cc:288] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 election: Requested vote from peers 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769), 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253:38113)
I20260929 17:48:12.577536 11241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:12.577901 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.577687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60778)
0929 17:48:12.577784 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.577835 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 17:48:12.577889 11241 leader_election.cc:288] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [CANDIDATE]: Term 2 election: Requested vote from peers 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769), 13a880bc74694fdd9afca5e092befe0d (127.3.183.252:44775)
I20260929 17:48:12.578073 10536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13a880bc74694fdd9afca5e092befe0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
I20260929 17:48:12.578205 10536 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 13a880bc74694fdd9afca5e092befe0d in current term 2: Already voted for candidate 8390d94a4ac14b62a5abb1b0cf25f68c in this term.
I20260929 17:48:12.578204 10729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13a880bc74694fdd9afca5e092befe0d"
I20260929 17:48:12.578349 10729 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8390d94a4ac14b62a5abb1b0cf25f68c in current term 2: Already voted for candidate 13a880bc74694fdd9afca5e092befe0d in this term.
I20260929 17:48:12.578559 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.578322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60790)
0929 17:48:12.578443 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.578488 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 17:48:12.578635 10704 leader_election.cc:334] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769): Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.578668 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.577968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60780)
0929 17:48:12.578437 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.578600 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
I20260929 17:48:12.578689 10704 leader_election.cc:302] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13a880bc74694fdd9afca5e092befe0d; no voters: 8390d94a4ac14b62a5abb1b0cf25f68c, 8b694f0ceec04751a088dc6d859267bb
I20260929 17:48:12.578809 11242 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 17:48:12.579214 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:12.579039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:60792)
0929 17:48:12.579127 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:12.579169 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:48:12.579303 10513 leader_election.cc:334] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769): Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:12.579370 10513 leader_election.cc:302] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8390d94a4ac14b62a5abb1b0cf25f68c; no voters: 13a880bc74694fdd9afca5e092befe0d, 8b694f0ceec04751a088dc6d859267bb
I20260929 17:48:12.579483 11241 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 17:48:13.061285 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:13.060879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:55901)
0929 17:48:13.061135 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:13.061193 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260929 17:48:13.062282 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:13.061810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:51485)
0929 17:48:13.062113 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:13.062198 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20260929 17:48:13.062393 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:13.062002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:53595)
0929 17:48:13.062240 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:13.062302 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":56,"threads_started":1}
I20260929 17:48:14.062383 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.061962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:40297)
0929 17:48:14.062204 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.062273 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260929 17:48:14.063264 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.062876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:41211)
0929 17:48:14.063107 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.063167 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260929 17:48:14.063540 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.063118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:58711)
0929 17:48:14.063372 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.063437 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260929 17:48:14.284498 11257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:14.284647 11257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:14.285058 11257 leader_election.cc:288] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769), 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253:38113)
I20260929 17:48:14.285650 10536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13a880bc74694fdd9afca5e092befe0d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" is_pre_election: true
I20260929 17:48:14.285784 10536 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a880bc74694fdd9afca5e092befe0d in term 2.
I20260929 17:48:14.285794 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.285303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:32806)
0929 17:48:14.285629 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.285691 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":90,"threads_started":1}
I20260929 17:48:14.286103 10703 leader_election.cc:302] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a880bc74694fdd9afca5e092befe0d, 8390d94a4ac14b62a5abb1b0cf25f68c; no voters:
I20260929 17:48:14.286283 11257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 17:48:14.286337 11257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:14.286374 11257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:48:14.287336 11257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:14.287637 11257 leader_election.cc:288] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 3 election: Requested vote from peers 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769), 8390d94a4ac14b62a5abb1b0cf25f68c (127.3.183.253:38113)
I20260929 17:48:14.287961 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.287743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:32810)
0929 17:48:14.287852 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.287900 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 17:48:14.288021 10536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13a880bc74694fdd9afca5e092befe0d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c"
I20260929 17:48:14.288152 10536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 2 FOLLOWER]: Advancing to term 3
W20260929 17:48:14.288658 10704 leader_election.cc:334] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769): Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:14.289079 10536 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a880bc74694fdd9afca5e092befe0d in term 3.
I20260929 17:48:14.289158 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.288939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:32818)
0929 17:48:14.289050 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.289103 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 17:48:14.289271 10704 leader_election.cc:334] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769): Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
I20260929 17:48:14.289371 10703 leader_election.cc:302] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13a880bc74694fdd9afca5e092befe0d, 8390d94a4ac14b62a5abb1b0cf25f68c; no voters: 8b694f0ceec04751a088dc6d859267bb
I20260929 17:48:14.289544 11257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 3 FOLLOWER]: Leader election won for term 3
I20260929 17:48:14.289723 11257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [term 3 LEADER]: Becoming Leader. State: Replica: 13a880bc74694fdd9afca5e092befe0d, State: Running, Role: LEADER
I20260929 17:48:14.289969 11257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } }
I20260929 17:48:14.290917 11260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13a880bc74694fdd9afca5e092befe0d. Latest consensus state: current_term: 3 leader_uuid: "13a880bc74694fdd9afca5e092befe0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:14.291028 11260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:14.291313 11262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:14.293577 11262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:14.293939 11262 catalog_manager.cc:1259] Loaded cluster ID: 4cc877435e54456b945bc782e7d97106
I20260929 17:48:14.293982 11262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:14.294495 11262 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:14.294838 11262 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Loaded TSK: 0
I20260929 17:48:14.295467 11262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:48:14.317401 10719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:14.321247 10719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:14.324141 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.323896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:32830)
0929 17:48:14.324007 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.324059 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 17:48:14.324299 10536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [term 3 FOLLOWER]: Refusing update from remote peer 13a880bc74694fdd9afca5e092befe0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 17:48:14.324867 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.324589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.0.0.1:32846)
0929 17:48:14.324697 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.324750 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 17:48:14.324765 11260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 17:48:14.325058 10704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d -> Peer 8b694f0ceec04751a088dc6d859267bb (127.3.183.254:45769): Couldn't send request to peer 8b694f0ceec04751a088dc6d859267bb. Status: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 17:48:14.326195 11264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13a880bc74694fdd9afca5e092befe0d. Latest consensus state: current_term: 3 leader_uuid: "13a880bc74694fdd9afca5e092befe0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:14.326314 11264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:48:14.327266 11257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "13a880bc74694fdd9afca5e092befe0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:14.327359 11257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:14.327606 11264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "13a880bc74694fdd9afca5e092befe0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b694f0ceec04751a088dc6d859267bb" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45769 } } peers { permanent_uuid: "8390d94a4ac14b62a5abb1b0cf25f68c" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 38113 } attrs { promote: false } } peers { permanent_uuid: "13a880bc74694fdd9afca5e092befe0d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 44775 } attrs { promote: false } } }
I20260929 17:48:14.327713 11264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8390d94a4ac14b62a5abb1b0cf25f68c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:48:14.336947 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.332649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:44667 (local address 127.0.0.1:42174)
0929 17:48:14.332746 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.332760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:14.332840 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:14.333270 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:14.333273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:14.333289 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:14.333412 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:14.333418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.334464 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.334467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:14.335661 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:14.335671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.336555 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.336558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:14.336652 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:14.336676 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:48:14.336753 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 17:48:14.336825 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 17:48:14.337008 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.332774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.3.183.195:44667)
0929 17:48:14.333034 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:14.333039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:14.333052 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:14.333102 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:14.333105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:14.333152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:14.333525 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:14.333634 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.334356 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.335770 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.336402 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.336553 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:14.336616 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:14.336781 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 17:48:14.336858 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260929 17:48:14.337679 11104 tablet_service.cc:1467] Processing CreateTablet for tablet 6d42dc870140409bab529261d1999f71 (DEFAULT_TABLE table=test-table [id=3c4d407a28bd476b9cfa8be8648511bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:14.337960 11104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d42dc870140409bab529261d1999f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:14.338002 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.333115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:37975 (local address 127.0.0.1:58580)
0929 17:48:14.333610 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.333623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:14.333695 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:14.334605 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:14.334607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:14.334622 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:14.334693 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:14.334697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.335620 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.335622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:14.336727 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:14.336736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.337678 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.337681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:14.337748 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:14.337764 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 17:48:14.337824 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:48:14.337893 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":87,"threads_started":1}
I20260929 17:48:14.338354 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.333376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.3.183.194:37975)
0929 17:48:14.334399 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:14.334402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:14.334415 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:14.334463 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:14.334466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:14.334510 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:14.334585 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:14.334779 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.335521 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.336856 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.337480 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.337579 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:14.337631 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:14.338143 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 17:48:14.338226 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":71,"threads_started":1}
I20260929 17:48:14.338842 10967 tablet_service.cc:1467] Processing CreateTablet for tablet 6d42dc870140409bab529261d1999f71 (DEFAULT_TABLE table=test-table [id=3c4d407a28bd476b9cfa8be8648511bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:14.339078 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.333197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39299 (local address 127.0.0.1:37508)
0929 17:48:14.334148 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.334158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:14.334233 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:14.335759 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:14.335761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:14.335811 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:14.335901 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:14.335907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.336823 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.336826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:14.337912 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:14.337921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.338705 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.338708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:14.338803 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:14.338826 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:48:14.338893 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 17:48:14.338957 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260929 17:48:14.339094 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d42dc870140409bab529261d1999f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:14.339126 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.333498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.3.183.193:39299)
0929 17:48:14.334037 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:14.334040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:14.334230 (+ 190us) server_negotiation.cc:407] Connection header received
0929 17:48:14.334308 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:14.334311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:14.334357 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:14.334440 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:14.336029 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.336716 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.338044 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.338598 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.338690 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:14.338744 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:14.338917 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 17:48:14.339004 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":71,"threads_started":1}
I20260929 17:48:14.339615 10830 tablet_service.cc:1467] Processing CreateTablet for tablet 6d42dc870140409bab529261d1999f71 (DEFAULT_TABLE table=test-table [id=3c4d407a28bd476b9cfa8be8648511bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:14.339865 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d42dc870140409bab529261d1999f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:14.341391 11276 tablet_bootstrap.cc:492] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: Bootstrap starting.
I20260929 17:48:14.341502 11275 tablet_bootstrap.cc:492] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: Bootstrap starting.
I20260929 17:48:14.342229 11275 tablet_bootstrap.cc:654] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:14.342283 11276 tablet_bootstrap.cc:654] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:14.342491 11277 tablet_bootstrap.cc:492] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110: Bootstrap starting.
I20260929 17:48:14.343371 11277 tablet_bootstrap.cc:654] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:14.343379 11275 tablet_bootstrap.cc:492] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: No bootstrap required, opened a new log
I20260929 17:48:14.343456 11275 ts_tablet_manager.cc:1397] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:14.343596 11276 tablet_bootstrap.cc:492] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: No bootstrap required, opened a new log
I20260929 17:48:14.343672 11276 ts_tablet_manager.cc:1397] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:14.343890 11275 raft_consensus.cc:348] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.343999 11275 raft_consensus.cc:374] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:14.344030 11275 raft_consensus.cc:729] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32093fe7d284f5bb9717e285bf918da, State: Initialized, Role: FOLLOWER
I20260929 17:48:14.344139 11276 raft_consensus.cc:348] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.344174 11275 consensus_queue.cc:260] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.344235 11276 raft_consensus.cc:374] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:14.344257 11276 raft_consensus.cc:729] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f0d93473db461695d14be89edf0640, State: Initialized, Role: FOLLOWER
I20260929 17:48:14.344352 11276 consensus_queue.cc:260] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.344528 11275 ts_tablet_manager.cc:1428] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:14.344959 11276 ts_tablet_manager.cc:1428] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:48:14.345212 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.344801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:39673)
0929 17:48:14.345092 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.345148 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260929 17:48:14.345558 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.345319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:40257)
0929 17:48:14.345456 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.345499 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 17:48:14.345924 11277 tablet_bootstrap.cc:492] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110: No bootstrap required, opened a new log
I20260929 17:48:14.346052 11277 ts_tablet_manager.cc:1397] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 17:48:14.346490 11277 raft_consensus.cc:348] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.346577 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.346352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:35851)
0929 17:48:14.346474 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.346519 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 17:48:14.346773 11170 heartbeater.cc:503] Master 127.3.183.252:44775 was elected leader, sending a full tablet report...
I20260929 17:48:14.347065 11277 raft_consensus.cc:374] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:14.347121 11277 raft_consensus.cc:729] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b7278a49834eba9417ddb666333110, State: Initialized, Role: FOLLOWER
I20260929 17:48:14.347262 11277 consensus_queue.cc:260] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.347450 11033 heartbeater.cc:503] Master 127.3.183.252:44775 was elected leader, sending a full tablet report...
I20260929 17:48:14.347627 11277 ts_tablet_manager.cc:1428] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:48:14.348032 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.347818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:35707)
0929 17:48:14.347916 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.347968 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 17:48:14.348264 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.347995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:45233)
0929 17:48:14.348132 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.348200 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 17:48:14.350034 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.349552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:60511)
0929 17:48:14.349928 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.349976 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316}
I20260929 17:48:14.350415 10896 heartbeater.cc:503] Master 127.3.183.252:44775 was elected leader, sending a full tablet report...
I20260929 17:48:14.383224 11283 raft_consensus.cc:482] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:14.383385 11283 raft_consensus.cc:504] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.383808 11283 leader_election.cc:288] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667), d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975)
I20260929 17:48:14.384177 11124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d42dc870140409bab529261d1999f71" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32093fe7d284f5bb9717e285bf918da" is_pre_election: true
I20260929 17:48:14.384177 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d42dc870140409bab529261d1999f71" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f0d93473db461695d14be89edf0640" is_pre_election: true
I20260929 17:48:14.384305 11124 raft_consensus.cc:2454] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 0.
I20260929 17:48:14.384316 10987 raft_consensus.cc:2454] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 0.
I20260929 17:48:14.384621 10783 leader_election.cc:302] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7b7278a49834eba9417ddb666333110, d7f0d93473db461695d14be89edf0640; no voters:
I20260929 17:48:14.384827 11283 raft_consensus.cc:2781] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:14.384872 11283 raft_consensus.cc:482] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:14.384904 11283 raft_consensus.cc:3037] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:14.385911 11283 raft_consensus.cc:504] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.386214 11283 leader_election.cc:288] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 election: Requested vote from peers a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667), d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975)
I20260929 17:48:14.386544 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d42dc870140409bab529261d1999f71" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f0d93473db461695d14be89edf0640"
I20260929 17:48:14.386545 11124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d42dc870140409bab529261d1999f71" candidate_uuid: "a7b7278a49834eba9417ddb666333110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32093fe7d284f5bb9717e285bf918da"
I20260929 17:48:14.386659 11124 raft_consensus.cc:3037] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:14.386659 10987 raft_consensus.cc:3037] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:14.387738 11124 raft_consensus.cc:2454] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 1.
I20260929 17:48:14.387733 10987 raft_consensus.cc:2454] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7b7278a49834eba9417ddb666333110 in term 1.
I20260929 17:48:14.388011 10783 leader_election.cc:302] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7b7278a49834eba9417ddb666333110, d7f0d93473db461695d14be89edf0640; no voters:
I20260929 17:48:14.388170 11283 raft_consensus.cc:2781] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:14.388379 11283 raft_consensus.cc:686] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [term 1 LEADER]: Becoming Leader. State: Replica: a7b7278a49834eba9417ddb666333110, State: Running, Role: LEADER
I20260929 17:48:14.388708 11283 consensus_queue.cc:237] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } }
I20260929 17:48:14.389094 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.388823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:39163)
0929 17:48:14.388961 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.389018 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 17:48:14.390198 10719 catalog_manager.cc:5495] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7b7278a49834eba9417ddb666333110 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "a7b7278a49834eba9417ddb666333110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7b7278a49834eba9417ddb666333110" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:14.400993 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.398220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.3.183.193:39299)
0929 17:48:14.398316 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:14.398319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:14.398331 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:48:14.398399 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:14.398401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:14.398438 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:14.398531 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:14.398705 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.399196 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.399973 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:14.400179 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:14.400254 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:14.400584 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:14.400653 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:14.400721 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:14.400849 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 17:48:14.400894 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 17:48:14.402872 11124 raft_consensus.cc:1261] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Refusing update from remote peer a7b7278a49834eba9417ddb666333110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:14.402952 10987 raft_consensus.cc:1261] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Refusing update from remote peer a7b7278a49834eba9417ddb666333110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:14.403337 11284 consensus_queue.cc:1035] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f0d93473db461695d14be89edf0640" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:14.403476 11283 consensus_queue.cc:1035] T 6d42dc870140409bab529261d1999f71 P a7b7278a49834eba9417ddb666333110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32093fe7d284f5bb9717e285bf918da" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:14.405396 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.405136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:35079)
0929 17:48:14.405260 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.405321 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 17:48:14.405491 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.405123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:50915)
0929 17:48:14.405379 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.405429 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260929 17:48:14.409013 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.408761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:35827)
0929 17:48:14.408877 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.408944 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:48:14.409140 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.408859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.194:47197)
0929 17:48:14.409025 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.409076 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 17:48:14.409610 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.409409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.193:51221)
0929 17:48:14.409503 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.409548 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 17:48:14.412166 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.411876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45769 (local address 127.3.183.195:51857)
0929 17:48:14.412058 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.412105 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 17:48:14.413630 10719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:14.413754 10719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:14.417001 10719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13a880bc74694fdd9afca5e092befe0d: Sending DeleteTablet for 3 replicas of tablet 6d42dc870140409bab529261d1999f71
I20260929 17:48:14.417531 11104 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d42dc870140409bab529261d1999f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:14 UTC) from {username='oryx', principal='oryx/127.3.183.252@KRBTEST.COM'} at 127.0.0.1:42174
I20260929 17:48:14.417699 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d42dc870140409bab529261d1999f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:14 UTC) from {username='oryx', principal='oryx/127.3.183.252@KRBTEST.COM'} at 127.0.0.1:58580
I20260929 17:48:14.417845 11295 tablet_replica.cc:330] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: stopping tablet replica
I20260929 17:48:14.417959 11295 raft_consensus.cc:2229] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:14.417951 10830 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d42dc870140409bab529261d1999f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:14 UTC) from {username='oryx', principal='oryx/127.3.183.252@KRBTEST.COM'} at 127.0.0.1:37508
I20260929 17:48:14.418040 11296 tablet_replica.cc:330] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: stopping tablet replica
I20260929 17:48:14.418074 11295 raft_consensus.cc:2258] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:14.418133 11296 raft_consensus.cc:2229] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:14.418354 11296 raft_consensus.cc:2258] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:14.418476 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 10670
I20260929 17:48:14.418684 11295 ts_tablet_manager.cc:1905] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:14.419178 11296 ts_tablet_manager.cc:1905] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:14.421407 11295 ts_tablet_manager.cc:1918] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:14.421499 11295 log.cc:1196] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/6d42dc870140409bab529261d1999f71
I20260929 17:48:14.421497 11296 ts_tablet_manager.cc:1918] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:14.421550 11296 log.cc:1196] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/6d42dc870140409bab529261d1999f71
I20260929 17:48:14.421751 11295 ts_tablet_manager.cc:1939] T 6d42dc870140409bab529261d1999f71 P a32093fe7d284f5bb9717e285bf918da: Deleting consensus metadata
I20260929 17:48:14.421777 11296 ts_tablet_manager.cc:1939] T 6d42dc870140409bab529261d1999f71 P d7f0d93473db461695d14be89edf0640: Deleting consensus metadata
I20260929 17:48:14.422377 10701 catalog_manager.cc:4841] TS d7f0d93473db461695d14be89edf0640 (127.3.183.194:37975): tablet 6d42dc870140409bab529261d1999f71 (table test-table [id=3c4d407a28bd476b9cfa8be8648511bd]) successfully deleted
I20260929 17:48:14.422438 10703 catalog_manager.cc:4841] TS a32093fe7d284f5bb9717e285bf918da (127.3.183.195:44667): tablet 6d42dc870140409bab529261d1999f71 (table test-table [id=3c4d407a28bd476b9cfa8be8648511bd]) successfully deleted
I20260929 17:48:14.426192 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 10904
I20260929 17:48:14.427158 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:14.426434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39299 (local address 127.0.0.1:37518)
0929 17:48:14.427019 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:14.427075 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:39299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493}
W20260929 17:48:14.427349 10702 catalog_manager.cc:4584] TS a7b7278a49834eba9417ddb666333110 (127.3.183.193:39299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d42dc870140409bab529261d1999f71: Network error: Client connection negotiation failed: client connection to 127.3.183.193:39299: connect: Connection refused (error 111)
I20260929 17:48:14.431314 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 11040
I20260929 17:48:14.435950 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 10390
I20260929 17:48:14.440699 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 10497
2026-09-29T17:48:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6976 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:14 dist-test-slave-wbpt krb5kdc[11300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:14 dist-test-slave-wbpt krb5kdc[11300](info): set up 2 sockets
Sep 29 17:48:14 dist-test-slave-wbpt krb5kdc[11300](info): commencing operation
krb5kdc: starting...
W20260929 17:48:15.396245 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:15 dist-test-slave-wbpt krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab.
Entry for principal oryx/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab.
I20260929 17:48:15.514377 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:39773
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab
--principal=oryx/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:15.620667 11313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:15.620927 11313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:15.620982 11313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:15.624936 11313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:15.624993 11313 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:48:15.625018 11313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:15.625041 11313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:15.625063 11313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:15.625090 11313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:15.630192 11313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:39773
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.254.keytab
--principal=oryx/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:15.631181 11313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:15.632200 11313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:15.640535 11321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:15.640658 11313 server_base.cc:1034] running on GCE node
W20260929 17:48:15.640501 11319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:15.640558 11318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:15.641295 11313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:15.642390 11313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:15.643567 11313 hybrid_clock.cc:648] HybridClock initialized: now 1790704095643549 us; error 28 us; skew 500 ppm
Sep 29 17:48:15 dist-test-slave-wbpt krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704095, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:15.646242 11313 init.cc:377] Logged in from keytab as oryx/127.3.183.254@KRBTEST.COM (short username oryx)
I20260929 17:48:15.647126 11313 webserver.cc:460] Webserver started at http://127.3.183.254:39537/ using document root <none> and password file <none>
I20260929 17:48:15.647655 11313 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:15.647718 11313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:15.647945 11313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:15.649981 11313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "2f89ba88fc304dad9b61f07bea5a2312"
format_stamp: "Formatted at 2026-09-29 17:48:15 on dist-test-slave-wbpt"
server_key: "a5f9a208ea66426bf6089661ee4786eb"
server_key_iv: "1aa80d8d61264c54338b42c5a142ec77"
server_key_version: "encryptionkey@0"
I20260929 17:48:15.650513 11313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "2f89ba88fc304dad9b61f07bea5a2312"
format_stamp: "Formatted at 2026-09-29 17:48:15 on dist-test-slave-wbpt"
server_key: "a5f9a208ea66426bf6089661ee4786eb"
server_key_iv: "1aa80d8d61264c54338b42c5a142ec77"
server_key_version: "encryptionkey@0"
I20260929 17:48:15.654176 11313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:15.656747 11328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:15.658042 11313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:15.658164 11313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "2f89ba88fc304dad9b61f07bea5a2312"
format_stamp: "Formatted at 2026-09-29 17:48:15 on dist-test-slave-wbpt"
server_key: "a5f9a208ea66426bf6089661ee4786eb"
server_key_iv: "1aa80d8d61264c54338b42c5a142ec77"
server_key_version: "encryptionkey@0"
I20260929 17:48:15.658268 11313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:15.673975 11313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:15.676944 11313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:15.677091 11313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:15.684182 11313 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:39773
I20260929 17:48:15.684224 11390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:39773 every 8 connection(s)
I20260929 17:48:15.685082 11313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:15.687764 11391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:15.690492 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 11313
I20260929 17:48:15.690621 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:15.690949 3807 external_mini_cluster.cc:1428] Setting key 8fd38822c04c6841dc22bc4bc46dacc1
I20260929 17:48:15.693676 11391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Bootstrap starting.
I20260929 17:48:15.696138 11391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:15.696990 11391 log.cc:824] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:15.698870 11391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: No bootstrap required, opened a new log
Sep 29 17:48:15 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:15.702323 11391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f89ba88fc304dad9b61f07bea5a2312" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 39773 } }
I20260929 17:48:15.702555 11391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:15.702592 11391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f89ba88fc304dad9b61f07bea5a2312, State: Initialized, Role: FOLLOWER
I20260929 17:48:15.703001 11391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f89ba88fc304dad9b61f07bea5a2312" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 39773 } }
I20260929 17:48:15.703127 11391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:15.703210 11391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:15.703320 11391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:15.704413 11391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f89ba88fc304dad9b61f07bea5a2312" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 39773 } }
I20260929 17:48:15.704662 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:15.692324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.3.183.254:39773)
0929 17:48:15.692721 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:15.692735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:15.692771 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:48:15.693355 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:15.693371 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:15.693709 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:15.694006 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:15.694842 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.695860 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.696956 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.697456 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.697606 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:15.699919 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:15.699924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:15.699938 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:15.699964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:15.701554 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:15.701992 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:15.701997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:15.702001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:15.702123 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:15.702383 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:15.702386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:15.702387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:15.702849 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:15.703066 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:15.703338 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 17:48:15.703637 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":100,"threads_started":1}
I20260929 17:48:15.704780 11391 leader_election.cc:302] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f89ba88fc304dad9b61f07bea5a2312; no voters:
I20260929 17:48:15.705096 11391 leader_election.cc:288] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:15.705233 11396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:15.705431 11396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [term 1 LEADER]: Becoming Leader. State: Replica: 2f89ba88fc304dad9b61f07bea5a2312, State: Running, Role: LEADER
I20260929 17:48:15.705786 11396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f89ba88fc304dad9b61f07bea5a2312" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 39773 } }
I20260929 17:48:15.706346 11391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:15.707386 11397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f89ba88fc304dad9b61f07bea5a2312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f89ba88fc304dad9b61f07bea5a2312" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 39773 } } }
I20260929 17:48:15.707509 11397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:15.707918 11398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f89ba88fc304dad9b61f07bea5a2312. Latest consensus state: current_term: 1 leader_uuid: "2f89ba88fc304dad9b61f07bea5a2312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f89ba88fc304dad9b61f07bea5a2312" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 39773 } } }
I20260929 17:48:15.708530 11398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312 [sys.catalog]: This master's current role is: LEADER
W20260929 17:48:15.710884 11412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:48:15.710948 11412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:48:15.711138 11405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:15.712172 11405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:15.717796 11405 catalog_manager.cc:1347] Generated new cluster ID: 39ccfbdc980d4d3bbb5b654f107471b8
I20260929 17:48:15.717877 11405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:15.734640 11405 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:15.735991 11405 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:15.744207 11405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Generated new TSK 0
I20260929 17:48:15.744897 11405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab.
Entry for principal oryx/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab.
I20260929 17:48:15.783067 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:39773
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab
--principal=oryx/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:15.903859 11417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:15.904121 11417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:15.904165 11417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:15.908202 11417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:15.908269 11417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:15.908316 11417 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:48:15.908349 11417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:15.913589 11417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab
--principal=oryx/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:15.914574 11417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:15.915521 11417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:15.924794 11422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:15.924773 11423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:15.924824 11417 server_base.cc:1034] running on GCE node
W20260929 17:48:15.924773 11425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:15.925426 11417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:15.926052 11417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:15.927232 11417 hybrid_clock.cc:648] HybridClock initialized: now 1790704095927208 us; error 33 us; skew 500 ppm
Sep 29 17:48:15 dist-test-slave-wbpt krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704095, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:15.929833 11417 init.cc:377] Logged in from keytab as oryx/127.3.183.193@KRBTEST.COM (short username oryx)
I20260929 17:48:15.930675 11417 webserver.cc:460] Webserver started at http://127.3.183.193:43705/ using document root <none> and password file <none>
I20260929 17:48:15.931193 11417 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:15.931257 11417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:15.931469 11417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:15.933445 11417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "b58cd9b60c444cea92a9ee7c417bf646"
format_stamp: "Formatted at 2026-09-29 17:48:15 on dist-test-slave-wbpt"
server_key: "df8a1930db55105b64c18f57b879ada8"
server_key_iv: "df43ae129468dd59b9ed0705858b3a3d"
server_key_version: "encryptionkey@0"
I20260929 17:48:15.933951 11417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "b58cd9b60c444cea92a9ee7c417bf646"
format_stamp: "Formatted at 2026-09-29 17:48:15 on dist-test-slave-wbpt"
server_key: "df8a1930db55105b64c18f57b879ada8"
server_key_iv: "df43ae129468dd59b9ed0705858b3a3d"
server_key_version: "encryptionkey@0"
I20260929 17:48:15.937463 11417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:15.939723 11432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:15.940896 11417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:15.941022 11417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "b58cd9b60c444cea92a9ee7c417bf646"
format_stamp: "Formatted at 2026-09-29 17:48:15 on dist-test-slave-wbpt"
server_key: "df8a1930db55105b64c18f57b879ada8"
server_key_iv: "df43ae129468dd59b9ed0705858b3a3d"
server_key_version: "encryptionkey@0"
I20260929 17:48:15.941118 11417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:15.967233 11417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:15.970111 11417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:15.970283 11417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:15.971679 11417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:15.971728 11417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:15.971809 11417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:15.971836 11417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:48:15 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704095, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:15.981218 11417 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:39781
I20260929 17:48:15.981287 11547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:39781 every 8 connection(s)
I20260929 17:48:15.982084 11417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:15.983896 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:15.972517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:50669 (local address 127.3.183.254:39773)
0929 17:48:15.972662 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:15.972667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:15.973516 (+ 849us) server_negotiation.cc:407] Connection header received
0929 17:48:15.974638 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:15.974642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:15.974701 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:15.974791 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:15.976188 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.976720 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.977872 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.978105 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.978196 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:15.980600 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:15.980605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:15.980607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:15.980631 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:15.982013 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:15.982518 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:15.982521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:15.982523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:15.982609 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:15.982967 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:15.982970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:15.982971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:15.983155 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:15.983234 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:15.983637 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 17:48:15.983732 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:48:15.984730 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:15.972914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:39773 (local address 127.3.183.193:50669)
0929 17:48:15.973354 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:15.973401 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:15.974367 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:15.974978 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:15.974986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:15.975391 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:15.976037 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:15.976053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.976833 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.976836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:15.977764 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:15.977772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.978252 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.978256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:15.978415 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:15.978934 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:15.978950 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:15.980437 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:15.982119 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:15.982125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:15.982139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:15.982404 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:15.982702 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:15.982706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:15.982708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:15.982864 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:15.983236 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:15.983241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:15.983451 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 17:48:15.983598 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 17:48:15.983867 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260929 17:48:15.985991 11548 heartbeater.cc:347] Connected to a master server at 127.3.183.254:39773
I20260929 17:48:15.986116 11548 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:15.986562 11548 heartbeater.cc:511] Master 127.3.183.254:39773 requested a full tablet report, sending...
I20260929 17:48:15.988085 11345 ts_manager.cc:194] Registered new tserver with Master: b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781)
I20260929 17:48:15.989445 11345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:50669
I20260929 17:48:15.989837 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 11417
I20260929 17:48:15.989955 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:15.990270 3807 external_mini_cluster.cc:1428] Setting key f5a0331af17f3a714eeba57d92538782
I20260929 17:48:15.995033 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:15.987984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:39773 (local address 127.3.183.193:55221)
0929 17:48:15.988340 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:15.988355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:15.988449 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:15.988667 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:15.988669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:15.988884 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:15.988997 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:15.989005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.989731 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.989734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:15.990464 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:15.990472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.991073 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.991077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:15.991167 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:15.991816 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:15.991831 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:15.992147 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:15.993825 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:15.993827 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:15.993829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:15.994009 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:15.994244 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:15.994246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:15.994247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:15.994310 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:15.994655 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:15.994657 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:15.994724 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 17:48:15.994793 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:48:15.994889 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":103,"threads_started":1}
I20260929 17:48:15.995091 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:15.988078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:55221 (local address 127.3.183.254:39773)
0929 17:48:15.988196 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:15.988200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:15.988421 (+ 221us) server_negotiation.cc:407] Connection header received
0929 17:48:15.988534 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:15.988536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:15.988579 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:15.988638 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:15.989093 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.989622 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.990573 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:15.990851 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:15.990936 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:15.992263 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:15.992266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:15.992269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:15.992291 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:15.993714 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:15.994087 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:15.994090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:15.994092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:15.994176 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:15.994373 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:15.994376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:15.994377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:15.994578 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:15.994661 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:15.994851 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 17:48:15.994955 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab.
Entry for principal oryx/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab.
I20260929 17:48:16.018472 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:39773
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab
--principal=oryx/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:16.126526 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:16.126776 11557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:16.126818 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:16.130697 11557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:16.130752 11557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:16.130789 11557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:48:16.130820 11557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:16.136086 11557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab
--principal=oryx/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:16.137066 11557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:16.138098 11557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:16.147914 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:16.147917 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:16.147967 11557 server_base.cc:1034] running on GCE node
W20260929 17:48:16.147917 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:16.148559 11557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:16.149188 11557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:16.150341 11557 hybrid_clock.cc:648] HybridClock initialized: now 1790704096150321 us; error 32 us; skew 500 ppm
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704096, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:16.153167 11557 init.cc:377] Logged in from keytab as oryx/127.3.183.194@KRBTEST.COM (short username oryx)
I20260929 17:48:16.154052 11557 webserver.cc:460] Webserver started at http://127.3.183.194:45621/ using document root <none> and password file <none>
I20260929 17:48:16.154608 11557 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:16.154667 11557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:16.154835 11557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:16.157012 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "8598767386d34aaf9fb66a24ab8d5d16"
format_stamp: "Formatted at 2026-09-29 17:48:16 on dist-test-slave-wbpt"
server_key: "38befd5ccde48fb3ea60e5b4763bc065"
server_key_iv: "5b0ced1055f9f58c8b13e06155c3296c"
server_key_version: "encryptionkey@0"
I20260929 17:48:16.157559 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "8598767386d34aaf9fb66a24ab8d5d16"
format_stamp: "Formatted at 2026-09-29 17:48:16 on dist-test-slave-wbpt"
server_key: "38befd5ccde48fb3ea60e5b4763bc065"
server_key_iv: "5b0ced1055f9f58c8b13e06155c3296c"
server_key_version: "encryptionkey@0"
I20260929 17:48:16.161283 11557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:48:16.163704 11572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:16.165009 11557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:16.165109 11557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "8598767386d34aaf9fb66a24ab8d5d16"
format_stamp: "Formatted at 2026-09-29 17:48:16 on dist-test-slave-wbpt"
server_key: "38befd5ccde48fb3ea60e5b4763bc065"
server_key_iv: "5b0ced1055f9f58c8b13e06155c3296c"
server_key_version: "encryptionkey@0"
I20260929 17:48:16.165216 11557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:16.181187 11557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:16.184319 11557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:16.184494 11557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:16.185890 11557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:16.185938 11557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:16.185976 11557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:16.186013 11557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704096, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:16.195648 11557 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:39463
I20260929 17:48:16.195731 11687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:39463 every 8 connection(s)
I20260929 17:48:16.196659 11557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:16.199752 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.186700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:44993 (local address 127.3.183.254:39773)
0929 17:48:16.186813 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.186816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.187938 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 17:48:16.189032 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.189036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.189096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.189190 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:16.190615 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.191423 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.192787 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.193108 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.193219 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.196021 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:16.196025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:16.196028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:16.196056 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:16.197522 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.198283 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.198286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.198287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.198366 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.198753 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.198755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.198757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.198932 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:16.199006 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.199523 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.199634 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:48:16.200510 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.187276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:39773 (local address 127.3.183.194:44993)
0929 17:48:16.187760 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.187837 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.188809 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.189371 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.189378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.189763 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:16.190395 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.190415 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.191555 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.191559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.192680 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.192689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.193244 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.193247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.193404 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.193979 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:16.194002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:16.195837 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:16.197619 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.197625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.197640 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.197989 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.198462 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.198465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.198466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.198611 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.199020 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:16.199025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:16.199251 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.199423 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.199680 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":145,"threads_started":1}
I20260929 17:48:16.201725 11688 heartbeater.cc:347] Connected to a master server at 127.3.183.254:39773
I20260929 17:48:16.201869 11688 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:16.202404 11688 heartbeater.cc:511] Master 127.3.183.254:39773 requested a full tablet report, sending...
I20260929 17:48:16.203426 11344 ts_manager.cc:194] Registered new tserver with Master: 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463)
I20260929 17:48:16.204173 11344 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:44993
I20260929 17:48:16.204666 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 11557
I20260929 17:48:16.204797 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:16.205119 3807 external_mini_cluster.cc:1428] Setting key 1294d776e7cea599c04acf9e5c11ea4f
I20260929 17:48:16.211380 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.203972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:35267 (local address 127.3.183.254:39773)
0929 17:48:16.204089 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.204093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.204312 (+ 219us) server_negotiation.cc:407] Connection header received
0929 17:48:16.204389 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.204391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.204457 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.204526 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:16.205107 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.205808 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.206829 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.207134 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.207258 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.208503 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:16.208506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:16.208508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:16.208529 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:16.209840 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.210316 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.210318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.210320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.210388 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.210649 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.210651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.210652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.210828 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:16.210897 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.211185 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.211271 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:48:16.211509 11694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.203926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:39773 (local address 127.3.183.194:35267)
0929 17:48:16.204215 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.204226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.204308 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.204645 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.204648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.204864 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:16.204985 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.204992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.205962 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.205966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.206718 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.206727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.207266 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.207269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.207345 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.207903 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:16.207921 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:16.208375 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:16.209950 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.209954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.209956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.210213 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.210466 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.210470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.210472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.210570 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.210917 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:16.210921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:16.211024 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.211140 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.211299 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab.
Entry for principal oryx/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab.
I20260929 17:48:16.230968 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:39773
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab
--principal=oryx/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:16.341418 11697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:16.341703 11697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:16.341765 11697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:16.345865 11697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:16.345927 11697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:16.345983 11697 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:48:16.346035 11697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:16.351300 11697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab
--principal=oryx/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.183.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:16.352393 11697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:16.353576 11697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:16.363615 11702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:16.363574 11705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:16.363838 11697 server_base.cc:1034] running on GCE node
W20260929 17:48:16.363715 11703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:16.364249 11697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:16.364854 11697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:16.366010 11697 hybrid_clock.cc:648] HybridClock initialized: now 1790704096365991 us; error 29 us; skew 500 ppm
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704096, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:16.368606 11697 init.cc:377] Logged in from keytab as oryx/127.3.183.195@KRBTEST.COM (short username oryx)
I20260929 17:48:16.369452 11697 webserver.cc:460] Webserver started at http://127.3.183.195:37571/ using document root <none> and password file <none>
I20260929 17:48:16.369946 11697 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:16.370002 11697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:16.370158 11697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:16.372123 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "dc4c61130a02403fae4118e7b15ceffc"
format_stamp: "Formatted at 2026-09-29 17:48:16 on dist-test-slave-wbpt"
server_key: "e2e778c45cc6fa21a114524fcb9ccfde"
server_key_iv: "c5304e4303f16a6a6abb3242d130267f"
server_key_version: "encryptionkey@0"
I20260929 17:48:16.372630 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "dc4c61130a02403fae4118e7b15ceffc"
format_stamp: "Formatted at 2026-09-29 17:48:16 on dist-test-slave-wbpt"
server_key: "e2e778c45cc6fa21a114524fcb9ccfde"
server_key_iv: "c5304e4303f16a6a6abb3242d130267f"
server_key_version: "encryptionkey@0"
I20260929 17:48:16.376118 11697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:16.378304 11712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:16.379406 11697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:16.379503 11697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "dc4c61130a02403fae4118e7b15ceffc"
format_stamp: "Formatted at 2026-09-29 17:48:16 on dist-test-slave-wbpt"
server_key: "e2e778c45cc6fa21a114524fcb9ccfde"
server_key_iv: "c5304e4303f16a6a6abb3242d130267f"
server_key_version: "encryptionkey@0"
I20260929 17:48:16.379590 11697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:16.418735 11697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:16.421665 11697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:16.421823 11697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:16.423226 11697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:16.423275 11697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:16.423313 11697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:16.423337 11697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704096, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:16.433202 11697 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:46623
I20260929 17:48:16.433249 11827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:46623 every 8 connection(s)
I20260929 17:48:16.434185 11697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:16.437108 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.424108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:53261 (local address 127.3.183.254:39773)
0929 17:48:16.424219 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.424223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.425319 (+ 1096us) server_negotiation.cc:407] Connection header received
0929 17:48:16.426409 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.426414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.426486 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.426547 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:16.428309 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.429027 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.430278 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.430590 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.430701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.433562 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:16.433566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:16.433568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:16.433597 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:16.435094 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.435619 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.435622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.435623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.435702 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.436116 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.436118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.436119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.436296 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:16.436369 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.436822 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.436939 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:48:16.437098 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 11697
I20260929 17:48:16.437229 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:16.437496 3807 external_mini_cluster.cc:1428] Setting key c8cd52ee76ecd00b8b3e7865e1b6e5f4
I20260929 17:48:16.438033 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.424682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:39773 (local address 127.3.183.195:53261)
0929 17:48:16.425165 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.425208 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.426153 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.426982 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.426991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.427413 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:16.428119 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.428140 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.429152 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.429156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.430153 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.430162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.430748 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.430753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.430936 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.431551 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:16.431573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:16.433394 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:16.435199 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.435204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.435214 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.435530 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.435848 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.435851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.435854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.436024 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.436373 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:16.436378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:16.436629 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.436781 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.437071 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
I20260929 17:48:16.440951 11828 heartbeater.cc:347] Connected to a master server at 127.3.183.254:39773
I20260929 17:48:16.441071 11828 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:16.441480 11828 heartbeater.cc:511] Master 127.3.183.254:39773 requested a full tablet report, sending...
I20260929 17:48:16.442414 11345 ts_manager.cc:194] Registered new tserver with Master: dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:16.443063 11345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:53261
I20260929 17:48:16.448203 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.441198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:56909 (local address 127.3.183.254:39773)
0929 17:48:16.441320 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.441324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.441547 (+ 223us) server_negotiation.cc:407] Connection header received
0929 17:48:16.441645 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.441648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.441690 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.441765 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:16.442144 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.442837 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.443855 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.444090 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.444168 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.445307 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:16.445310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:16.445312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:16.445332 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:16.446698 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.447082 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.447084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.447085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.447151 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.447406 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.447408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.447409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.447631 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:16.447706 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.447967 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.448065 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:48:16.448314 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.441199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:39773 (local address 127.3.183.195:56909)
0929 17:48:16.441462 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.441476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.441571 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.441776 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.441778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.441946 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:16.442038 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.442043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.442952 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.442955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.443716 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.443726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.444193 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.444197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.444283 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.444797 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:16.444810 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:16.445196 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:16.446800 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.446802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.446804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.447011 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.447246 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:16.447248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:16.447250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:16.447334 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:16.447702 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:16.447704 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:16.447773 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.447898 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.448180 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260929 17:48:16.451239 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:48:16.473413 11394 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:16.467732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.3.183.254:39773)
0929 17:48:16.467865 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.467868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.467881 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:16.467920 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.467922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.467960 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.468054 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:16.468727 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.469264 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.470087 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.470291 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.470355 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.473291 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54404: BlockingRecv error: recv got EOF from 127.0.0.1:54404 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 17:48:16 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:16.481694 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.474716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.3.183.254:39773)
0929 17:48:16.474825 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.474827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.474964 (+ 137us) server_negotiation.cc:407] Connection header received
0929 17:48:16.475058 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.475060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.475110 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.475171 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:16.475662 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.476220 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.476958 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.477183 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.477259 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.479332 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:16.479335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:16.479337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:16.479357 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:16.480499 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.480840 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.480842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.480843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.480905 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:16.481086 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:16.481088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:16.481089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:16.481232 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:16.481323 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.481475 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.481555 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:48:16.484300 11345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:16.487332 11345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:16.500926 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.495236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39781 (local address 127.0.0.1:59896)
0929 17:48:16.495691 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.495737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.495990 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.496758 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.496763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.496786 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:16.496930 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.496938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.498166 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.498169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.499608 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.499619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.500406 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.500410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.500522 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.500558 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.500663 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.500782 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":74,"thread_start_us":78,"threads_started":1}
I20260929 17:48:16.500926 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.495362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.3.183.193:39781)
0929 17:48:16.495756 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.495763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.495876 (+ 113us) server_negotiation.cc:407] Connection header received
0929 17:48:16.496191 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.496197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.496430 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.496582 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:16.497054 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.498066 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.499729 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.500302 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.500397 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.500545 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.500699 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.500792 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":82,"threads_started":1}
I20260929 17:48:16.501123 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.495367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39463 (local address 127.0.0.1:44370)
0929 17:48:16.496059 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.496093 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.496180 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.496967 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.496970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.496984 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:16.497067 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.497072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.498494 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.498497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.499638 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.499645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.500746 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.500750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.500836 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.500859 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.500908 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.500976 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":19,"thread_start_us":89,"threads_started":1}
I20260929 17:48:16.501686 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.495953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:46623 (local address 127.0.0.1:42014)
0929 17:48:16.496826 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.496838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.496930 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.497389 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.497391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.497408 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:16.497507 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.497513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.498749 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.498753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.499758 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.499767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.500924 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.500927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.501417 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.501438 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.501503 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.501569 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":29,"thread_start_us":87,"threads_started":1}
I20260929 17:48:16.501824 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.496136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.3.183.194:39463)
0929 17:48:16.496524 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.496535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.496557 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:48:16.496642 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.496647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.496832 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.497191 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:16.497286 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.498377 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.499753 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.500613 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.501393 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.501568 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.501616 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.501699 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":108,"threads_started":1}
I20260929 17:48:16.502090 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.496352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.3.183.195:46623)
0929 17:48:16.496690 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.496700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.496916 (+ 216us) server_negotiation.cc:407] Connection header received
0929 17:48:16.497050 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.497055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.497249 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.497396 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:16.497636 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.498661 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.499878 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.500613 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.500717 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.500871 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.501854 (+ 983us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.501952 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":75,"threads_started":1}
I20260929 17:48:16.503746 11482 tablet_service.cc:1467] Processing CreateTablet for tablet 90f2c73515a2490999b3879cc5c8d746 (DEFAULT_TABLE table=test-table [id=695a9dcf0c4c4c8db188424dea376278]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:16.504820 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f2c73515a2490999b3879cc5c8d746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:16.506466 11762 tablet_service.cc:1467] Processing CreateTablet for tablet 90f2c73515a2490999b3879cc5c8d746 (DEFAULT_TABLE table=test-table [id=695a9dcf0c4c4c8db188424dea376278]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:16.506374 11622 tablet_service.cc:1467] Processing CreateTablet for tablet 90f2c73515a2490999b3879cc5c8d746 (DEFAULT_TABLE table=test-table [id=695a9dcf0c4c4c8db188424dea376278]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:16.507287 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f2c73515a2490999b3879cc5c8d746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:16.507351 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f2c73515a2490999b3879cc5c8d746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:16.511979 11856 tablet_bootstrap.cc:492] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Bootstrap starting.
I20260929 17:48:16.514086 11858 tablet_bootstrap.cc:492] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Bootstrap starting.
I20260929 17:48:16.514212 11856 tablet_bootstrap.cc:654] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:16.514967 11857 tablet_bootstrap.cc:492] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Bootstrap starting.
I20260929 17:48:16.515035 11856 log.cc:824] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:16.516140 11858 tablet_bootstrap.cc:654] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:16.516724 11856 tablet_bootstrap.cc:492] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: No bootstrap required, opened a new log
I20260929 17:48:16.516898 11856 ts_tablet_manager.cc:1397] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 17:48:16.516924 11858 log.cc:824] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:16.516994 11857 tablet_bootstrap.cc:654] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:16.517669 11857 log.cc:824] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:16.518775 11858 tablet_bootstrap.cc:492] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: No bootstrap required, opened a new log
I20260929 17:48:16.518949 11858 ts_tablet_manager.cc:1397] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:48:16.519519 11857 tablet_bootstrap.cc:492] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: No bootstrap required, opened a new log
I20260929 17:48:16.519685 11857 ts_tablet_manager.cc:1397] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 17:48:16.520750 11856 raft_consensus.cc:348] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.520953 11856 raft_consensus.cc:374] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:16.521009 11856 raft_consensus.cc:729] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58cd9b60c444cea92a9ee7c417bf646, State: Initialized, Role: FOLLOWER
I20260929 17:48:16.521384 11856 consensus_queue.cc:260] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.522027 11548 heartbeater.cc:503] Master 127.3.183.254:39773 was elected leader, sending a full tablet report...
I20260929 17:48:16.522207 11856 ts_tablet_manager.cc:1428] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 17:48:16.523262 11857 raft_consensus.cc:348] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.523501 11857 raft_consensus.cc:374] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:16.523561 11857 raft_consensus.cc:729] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4c61130a02403fae4118e7b15ceffc, State: Initialized, Role: FOLLOWER
I20260929 17:48:16.523852 11858 raft_consensus.cc:348] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.523978 11857 consensus_queue.cc:260] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.524080 11858 raft_consensus.cc:374] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:16.524137 11858 raft_consensus.cc:729] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Initialized, Role: FOLLOWER
I20260929 17:48:16.524647 11828 heartbeater.cc:503] Master 127.3.183.254:39773 was elected leader, sending a full tablet report...
I20260929 17:48:16.524884 11857 ts_tablet_manager.cc:1428] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:48:16.525557 11858 consensus_queue.cc:260] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.526198 11688 heartbeater.cc:503] Master 127.3.183.254:39773 was elected leader, sending a full tablet report...
I20260929 17:48:16.526505 11858 ts_tablet_manager.cc:1428] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 17:48:16.685164 11829 tablet.cc:2369] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:48:16.697794 11689 tablet.cc:2369] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:48:16.733465 11549 tablet.cc:2369] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:16.863409 11862 raft_consensus.cc:482] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:16.863657 11862 raft_consensus.cc:504] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.864833 11862 leader_election.cc:288] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:16.868570 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.865192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:58477 (local address 127.3.183.194:39463)
0929 17:48:16.865318 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.865322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.865568 (+ 246us) server_negotiation.cc:407] Connection header received
0929 17:48:16.865697 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.865700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.865745 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.865807 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:16.866068 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.866563 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.867604 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.868061 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.868165 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.868216 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.868381 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.868443 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:48:16.868589 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.865188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:40891 (local address 127.3.183.195:46623)
0929 17:48:16.865318 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.865322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.865739 (+ 417us) server_negotiation.cc:407] Connection header received
0929 17:48:16.865816 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.865819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.865863 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.865928 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:16.866173 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.866668 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.867645 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.868068 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.868169 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.868217 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.868406 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.868470 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:48:16.868607 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.865080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39463 (local address 127.3.183.193:58477)
0929 17:48:16.865472 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.865484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.865602 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.865829 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.865832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.865847 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:16.865962 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.865967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.866665 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.866668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.867482 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.867491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.868165 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.868168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.868262 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.868303 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.868368 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.868445 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":136,"threads_started":1}
I20260929 17:48:16.868623 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.865096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:46623 (local address 127.3.183.193:40891)
0929 17:48:16.865614 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:16.865627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:16.865734 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:16.865944 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:16.865947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:16.865962 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:16.866069 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.866074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.866773 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.866776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.867527 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:16.867536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.868194 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.868198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:16.868298 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.868315 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:48:16.868377 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:48:16.868460 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"spinlock_wait_cycles":11008,"thread_start_us":119,"threads_started":1}
I20260929 17:48:16.869259 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f2c73515a2490999b3879cc5c8d746" candidate_uuid: "b58cd9b60c444cea92a9ee7c417bf646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc" is_pre_election: true
I20260929 17:48:16.869314 11642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f2c73515a2490999b3879cc5c8d746" candidate_uuid: "b58cd9b60c444cea92a9ee7c417bf646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8598767386d34aaf9fb66a24ab8d5d16" is_pre_election: true
I20260929 17:48:16.869547 11782 raft_consensus.cc:2454] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b58cd9b60c444cea92a9ee7c417bf646 in term 0.
I20260929 17:48:16.869570 11642 raft_consensus.cc:2454] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b58cd9b60c444cea92a9ee7c417bf646 in term 0.
I20260929 17:48:16.869966 11434 leader_election.cc:302] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b58cd9b60c444cea92a9ee7c417bf646, dc4c61130a02403fae4118e7b15ceffc; no voters:
I20260929 17:48:16.870213 11862 raft_consensus.cc:2781] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:16.870296 11862 raft_consensus.cc:482] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:16.870342 11862 raft_consensus.cc:3037] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:16.871294 11862 raft_consensus.cc:504] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.871579 11862 leader_election.cc:288] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [CANDIDATE]: Term 1 election: Requested vote from peers 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:16.871943 11642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f2c73515a2490999b3879cc5c8d746" candidate_uuid: "b58cd9b60c444cea92a9ee7c417bf646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8598767386d34aaf9fb66a24ab8d5d16"
I20260929 17:48:16.871955 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f2c73515a2490999b3879cc5c8d746" candidate_uuid: "b58cd9b60c444cea92a9ee7c417bf646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc"
I20260929 17:48:16.872051 11642 raft_consensus.cc:3037] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:16.872051 11782 raft_consensus.cc:3037] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:16.873050 11642 raft_consensus.cc:2454] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b58cd9b60c444cea92a9ee7c417bf646 in term 1.
I20260929 17:48:16.873322 11435 leader_election.cc:302] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, b58cd9b60c444cea92a9ee7c417bf646; no voters:
I20260929 17:48:16.873482 11862 raft_consensus.cc:2781] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:16.873538 11782 raft_consensus.cc:2454] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b58cd9b60c444cea92a9ee7c417bf646 in term 1.
I20260929 17:48:16.873663 11862 raft_consensus.cc:686] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 LEADER]: Becoming Leader. State: Replica: b58cd9b60c444cea92a9ee7c417bf646, State: Running, Role: LEADER
I20260929 17:48:16.874020 11862 consensus_queue.cc:237] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:16.876995 11341 catalog_manager.cc:5495] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 reported cstate change: term changed from 0 to 1, leader changed from <none> to b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "b58cd9b60c444cea92a9ee7c417bf646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:16.959139 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:16.956012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.3.183.193:39781)
0929 17:48:16.956149 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:16.956153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:16.956166 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:16.956264 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:16.956266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:16.956318 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:16.956403 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:16.956592 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.957092 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.957891 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:16.958112 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:16.958189 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:16.958390 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:16.958837 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:16.958918 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:16.958979 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 17:48:16.959032 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:48:16.964923 11782 raft_consensus.cc:1261] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Refusing update from remote peer b58cd9b60c444cea92a9ee7c417bf646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:16.964941 11642 raft_consensus.cc:1261] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Refusing update from remote peer b58cd9b60c444cea92a9ee7c417bf646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:16.965605 11867 consensus_queue.cc:1035] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:16.965780 11862 consensus_queue.cc:1035] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:16.975145 11871 mvcc.cc:205] Tried to move back new op lower bound from 7334723981161762816 to 7334723980798205952. Current Snapshot: MvccSnapshot[applied={T|T < 7334723981161762816}]
I20260929 17:48:16.981335 11345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:16.981518 11345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:16.984141 11345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Sending DeleteTablet for 3 replicas of tablet 90f2c73515a2490999b3879cc5c8d746
I20260929 17:48:16.984899 11762 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f2c73515a2490999b3879cc5c8d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:16 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260929 17:48:16.985344 11880 tablet_replica.cc:330] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: stopping tablet replica
I20260929 17:48:16.985491 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f2c73515a2490999b3879cc5c8d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:16 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:59896
I20260929 17:48:16.985737 11880 raft_consensus.cc:2229] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:16.986052 11880 raft_consensus.cc:2258] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:16.986671 11622 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f2c73515a2490999b3879cc5c8d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:16 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260929 17:48:16.987100 11882 tablet_replica.cc:330] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: stopping tablet replica
I20260929 17:48:16.987391 11882 raft_consensus.cc:2229] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:16.987665 11882 raft_consensus.cc:2258] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:16.987875 11345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:16.988266 11345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:16.990159 11882 ts_tablet_manager.cc:1905] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:16.992909 11882 ts_tablet_manager.cc:1918] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:16.993021 11882 log.cc:1196] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/90f2c73515a2490999b3879cc5c8d746
I20260929 17:48:16.993299 11882 ts_tablet_manager.cc:1939] T 90f2c73515a2490999b3879cc5c8d746 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting consensus metadata
I20260929 17:48:16.994019 11482 tablet_service.cc:1467] Processing CreateTablet for tablet 3b59550b04464d35b0c708b309048f33 (DEFAULT_TABLE table=test-table [id=dd1ab704785b458985a152d1269c93a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:16.994270 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b59550b04464d35b0c708b309048f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:16.996112 11762 tablet_service.cc:1467] Processing CreateTablet for tablet 3b59550b04464d35b0c708b309048f33 (DEFAULT_TABLE table=test-table [id=dd1ab704785b458985a152d1269c93a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:16.996219 11331 catalog_manager.cc:4841] TS 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463): tablet 90f2c73515a2490999b3879cc5c8d746 (table test-table [id=695a9dcf0c4c4c8db188424dea376278]) successfully deleted
I20260929 17:48:16.996410 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b59550b04464d35b0c708b309048f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:16.996570 11856 tablet_bootstrap.cc:492] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: Bootstrap starting.
I20260929 17:48:16.997531 11856 tablet_bootstrap.cc:654] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:16.998207 11880 ts_tablet_manager.cc:1905] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:16.998445 11622 tablet_service.cc:1467] Processing CreateTablet for tablet 3b59550b04464d35b0c708b309048f33 (DEFAULT_TABLE table=test-table [id=dd1ab704785b458985a152d1269c93a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:16.998710 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b59550b04464d35b0c708b309048f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:16.998760 11857 tablet_bootstrap.cc:492] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: Bootstrap starting.
I20260929 17:48:16.999653 11857 tablet_bootstrap.cc:654] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.000814 11880 ts_tablet_manager.cc:1918] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:17.000901 11880 log.cc:1196] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/90f2c73515a2490999b3879cc5c8d746
I20260929 17:48:17.001196 11880 ts_tablet_manager.cc:1939] T 90f2c73515a2490999b3879cc5c8d746 P dc4c61130a02403fae4118e7b15ceffc: Deleting consensus metadata
I20260929 17:48:17.001212 11857 tablet_bootstrap.cc:492] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: No bootstrap required, opened a new log
I20260929 17:48:17.001307 11857 ts_tablet_manager.cc:1397] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:48:17.001772 11857 raft_consensus.cc:348] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.001895 11857 raft_consensus.cc:374] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.001931 11857 raft_consensus.cc:729] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4c61130a02403fae4118e7b15ceffc, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.002059 11857 consensus_queue.cc:260] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.002326 11857 ts_tablet_manager.cc:1428] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.002367 11858 tablet_bootstrap.cc:492] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: Bootstrap starting.
I20260929 17:48:17.003172 11881 tablet_replica.cc:330] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: stopping tablet replica
I20260929 17:48:17.003331 11858 tablet_bootstrap.cc:654] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.003605 11881 raft_consensus.cc:2229] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:48:17.003993 11856 tablet_bootstrap.cc:492] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: No bootstrap required, opened a new log
I20260929 17:48:17.004089 11856 ts_tablet_manager.cc:1397] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260929 17:48:17.004181 11881 raft_consensus.cc:2258] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.004465 11330 catalog_manager.cc:4841] TS dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623): tablet 90f2c73515a2490999b3879cc5c8d746 (table test-table [id=695a9dcf0c4c4c8db188424dea376278]) successfully deleted
I20260929 17:48:17.004523 11858 tablet_bootstrap.cc:492] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: No bootstrap required, opened a new log
I20260929 17:48:17.004602 11858 ts_tablet_manager.cc:1397] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:17.004557 11856 raft_consensus.cc:348] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.004685 11856 raft_consensus.cc:374] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.004730 11856 raft_consensus.cc:729] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58cd9b60c444cea92a9ee7c417bf646, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.004864 11856 consensus_queue.cc:260] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.005123 11856 ts_tablet_manager.cc:1428] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.005313 11858 raft_consensus.cc:348] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.005422 11858 raft_consensus.cc:374] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.005452 11858 raft_consensus.cc:729] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.005575 11858 consensus_queue.cc:260] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.005826 11858 ts_tablet_manager.cc:1428] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.007649 11881 ts_tablet_manager.cc:1905] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.010465 11881 ts_tablet_manager.cc:1918] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:17.010589 11881 log.cc:1196] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/90f2c73515a2490999b3879cc5c8d746
I20260929 17:48:17.010943 11881 ts_tablet_manager.cc:1939] T 90f2c73515a2490999b3879cc5c8d746 P b58cd9b60c444cea92a9ee7c417bf646: Deleting consensus metadata
I20260929 17:48:17.012658 11330 catalog_manager.cc:4841] TS b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781): tablet 90f2c73515a2490999b3879cc5c8d746 (table test-table [id=695a9dcf0c4c4c8db188424dea376278]) successfully deleted
I20260929 17:48:17.036996 11863 raft_consensus.cc:482] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:17.037143 11863 raft_consensus.cc:504] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.038250 11863 leader_election.cc:288] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463)
I20260929 17:48:17.043273 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.038965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:52393 (local address 127.3.183.194:39463)
0929 17:48:17.039096 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.039100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.039294 (+ 194us) server_negotiation.cc:407] Connection header received
0929 17:48:17.039444 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.039448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.039493 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.039540 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:17.040033 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.040841 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.042253 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.042657 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.042767 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.042823 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.043023 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.043120 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:48:17.043366 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.038669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:48633 (local address 127.3.183.193:39781)
0929 17:48:17.038795 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.038800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.039239 (+ 439us) server_negotiation.cc:407] Connection header received
0929 17:48:17.039445 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.039448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.039493 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.039550 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:17.040054 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.040649 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.042224 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.042837 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.042985 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.043038 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.043167 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.043240 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:48:17.043642 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.038574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39463 (local address 127.3.183.195:52393)
0929 17:48:17.039227 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.039239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.039339 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.039677 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.039679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.039717 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:17.039916 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.039921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.040952 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.040955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.042104 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.042109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.042767 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.042770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.042879 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.042902 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.043436 (+ 534us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.043505 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":77,"spinlock_wait_cycles":28160,"thread_start_us":99,"threads_started":1}
I20260929 17:48:17.043757 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.038560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39781 (local address 127.3.183.195:48633)
0929 17:48:17.039149 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.039164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.039322 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.039677 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.039679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.039719 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:17.039913 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.039921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.040752 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.040756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.042096 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.042107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.042969 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.042973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.043060 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.043079 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.043142 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.043213 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":49,"spinlock_wait_cycles":3584,"thread_start_us":87,"threads_started":1}
I20260929 17:48:17.044147 11642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b59550b04464d35b0c708b309048f33" candidate_uuid: "dc4c61130a02403fae4118e7b15ceffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8598767386d34aaf9fb66a24ab8d5d16" is_pre_election: true
I20260929 17:48:17.044309 11642 raft_consensus.cc:2454] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4c61130a02403fae4118e7b15ceffc in term 0.
I20260929 17:48:17.044481 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b59550b04464d35b0c708b309048f33" candidate_uuid: "dc4c61130a02403fae4118e7b15ceffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646" is_pre_election: true
I20260929 17:48:17.044669 11715 leader_election.cc:302] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, dc4c61130a02403fae4118e7b15ceffc; no voters:
I20260929 17:48:17.044763 11502 raft_consensus.cc:2454] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4c61130a02403fae4118e7b15ceffc in term 0.
I20260929 17:48:17.045037 11863 raft_consensus.cc:2781] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:17.045104 11863 raft_consensus.cc:482] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:17.045145 11863 raft_consensus.cc:3037] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.046164 11863 raft_consensus.cc:504] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.046505 11863 leader_election.cc:288] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [CANDIDATE]: Term 1 election: Requested vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463)
I20260929 17:48:17.046836 11642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b59550b04464d35b0c708b309048f33" candidate_uuid: "dc4c61130a02403fae4118e7b15ceffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8598767386d34aaf9fb66a24ab8d5d16"
I20260929 17:48:17.046854 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b59550b04464d35b0c708b309048f33" candidate_uuid: "dc4c61130a02403fae4118e7b15ceffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646"
I20260929 17:48:17.046950 11642 raft_consensus.cc:3037] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.046975 11502 raft_consensus.cc:3037] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.048156 11642 raft_consensus.cc:2454] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc4c61130a02403fae4118e7b15ceffc in term 1.
I20260929 17:48:17.048358 11502 raft_consensus.cc:2454] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc4c61130a02403fae4118e7b15ceffc in term 1.
I20260929 17:48:17.048544 11715 leader_election.cc:302] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, dc4c61130a02403fae4118e7b15ceffc; no voters:
I20260929 17:48:17.048776 11863 raft_consensus.cc:2781] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:17.049119 11863 raft_consensus.cc:686] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 1 LEADER]: Becoming Leader. State: Replica: dc4c61130a02403fae4118e7b15ceffc, State: Running, Role: LEADER
I20260929 17:48:17.049384 11863 consensus_queue.cc:237] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.052045 11344 catalog_manager.cc:5495] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc reported cstate change: term changed from 0 to 1, leader changed from <none> to dc4c61130a02403fae4118e7b15ceffc (127.3.183.195). New cstate: current_term: 1 leader_uuid: "dc4c61130a02403fae4118e7b15ceffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } health_report { overall_health: HEALTHY } } }
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704095, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.254@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:17.098511 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.089514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:39773 (local address 127.0.0.1:54416)
0929 17:48:17.089820 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.089837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.089946 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.090287 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.090289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.090723 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:17.090830 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.090836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.091907 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.091910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.092461 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.092469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.093058 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.093061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.093132 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.093676 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:17.093688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:17.095199 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:17.097085 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:17.097095 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:17.097099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:17.097327 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:17.097603 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:17.097605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:17.097606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:17.097673 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:17.098081 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:17.098086 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:17.098185 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.098269 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.098374 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":104,"threads_started":1}
I20260929 17:48:17.098666 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.089597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.3.183.254:39773)
0929 17:48:17.089863 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.089871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.089891 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:48:17.090044 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.090047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.090101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.090229 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:17.090980 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.091766 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.092577 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.092944 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.093053 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.095335 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:17.095339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:17.095341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:17.095365 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:17.096971 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:17.097420 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:17.097424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:17.097425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:17.097517 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:17.097752 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:17.097755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:17.097757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:17.097991 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:17.098098 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.098348 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.098471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260929 17:48:17.102634 11344 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:54416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:48:17.103471 11344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:17.108584 11482 tablet_service.cc:1467] Processing CreateTablet for tablet 0faf8650ac64408a809da424f71329eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c11ec07ae0a4569a35598fdef068de4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:48:17.108826 11622 tablet_service.cc:1467] Processing CreateTablet for tablet 0faf8650ac64408a809da424f71329eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c11ec07ae0a4569a35598fdef068de4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:48:17.109010 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faf8650ac64408a809da424f71329eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.109161 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faf8650ac64408a809da424f71329eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.111819 11856 tablet_bootstrap.cc:492] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: Bootstrap starting.
I20260929 17:48:17.112239 11762 tablet_service.cc:1467] Processing CreateTablet for tablet 0faf8650ac64408a809da424f71329eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c11ec07ae0a4569a35598fdef068de4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:48:17.112600 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faf8650ac64408a809da424f71329eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.112879 11856 tablet_bootstrap.cc:654] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.113778 11858 tablet_bootstrap.cc:492] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: Bootstrap starting.
I20260929 17:48:17.114540 11858 tablet_bootstrap.cc:654] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.115322 11856 tablet_bootstrap.cc:492] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: No bootstrap required, opened a new log
I20260929 17:48:17.115383 11856 ts_tablet_manager.cc:1397] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:48:17.115527 11857 tablet_bootstrap.cc:492] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: Bootstrap starting.
I20260929 17:48:17.115835 11856 raft_consensus.cc:348] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.115952 11856 raft_consensus.cc:374] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.115993 11856 raft_consensus.cc:729] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58cd9b60c444cea92a9ee7c417bf646, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.116498 11858 tablet_bootstrap.cc:492] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: No bootstrap required, opened a new log
I20260929 17:48:17.116586 11858 ts_tablet_manager.cc:1397] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:48:17.116600 11857 tablet_bootstrap.cc:654] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.117002 11858 raft_consensus.cc:348] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.117110 11858 raft_consensus.cc:374] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.117141 11858 raft_consensus.cc:729] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.117115 11856 consensus_queue.cc:260] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.117270 11858 consensus_queue.cc:260] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.117513 11858 ts_tablet_manager.cc:1428] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.117452 11876 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.117664 11876 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:48:17.117432 11856 ts_tablet_manager.cc:1428] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:17.117817 11864 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.117951 11864 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:48:17.118965 11857 tablet_bootstrap.cc:492] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: No bootstrap required, opened a new log
I20260929 17:48:17.119055 11857 ts_tablet_manager.cc:1397] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:48:17.119463 11857 raft_consensus.cc:348] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.119552 11857 raft_consensus.cc:374] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.119580 11857 raft_consensus.cc:729] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4c61130a02403fae4118e7b15ceffc, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.119695 11857 consensus_queue.cc:260] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.119980 11857 ts_tablet_manager.cc:1428] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.120569 11863 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.120687 11863 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:48:17.175838 11864 raft_consensus.cc:482] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:17.175980 11864 raft_consensus.cc:504] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.177049 11864 leader_election.cc:288] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:17.182587 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.177803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:39465 (local address 127.3.183.195:46623)
0929 17:48:17.177918 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.177922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.178133 (+ 211us) server_negotiation.cc:407] Connection header received
0929 17:48:17.178211 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.178213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.178257 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.178321 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:17.178557 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.179372 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.181270 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.181942 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.182083 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.182144 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.182334 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.182456 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:48:17.182587 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.177377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:33773 (local address 127.3.183.193:39781)
0929 17:48:17.177512 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.177515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.177940 (+ 425us) server_negotiation.cc:407] Connection header received
0929 17:48:17.178144 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.178148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.178200 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.178278 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:17.178558 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.179371 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.181270 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.181942 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.182083 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.182144 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.182333 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.182444 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:48:17.182629 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.177698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:46623 (local address 127.3.183.194:39465)
0929 17:48:17.178056 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.178068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.178145 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.178334 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.178337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.178369 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:17.178460 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.178466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.179550 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.179554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.181117 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.181128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.182115 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.182118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.182202 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.182239 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.182310 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.182454 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":224,"spinlock_wait_cycles":384,"thread_start_us":84,"threads_started":1}
I20260929 17:48:17.182636 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.177286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39781 (local address 127.3.183.194:33773)
0929 17:48:17.177854 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.177868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.178016 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.178287 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.178290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.178304 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:17.178459 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.178466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.179550 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.179554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.181117 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.181128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.182074 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.182077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.182190 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.182236 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.182310 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.182451 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":114,"spinlock_wait_cycles":4608,"thread_start_us":133,"threads_started":1}
I20260929 17:48:17.183135 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf8650ac64408a809da424f71329eb" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646" is_pre_election: true
I20260929 17:48:17.183310 11502 raft_consensus.cc:2454] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 0.
I20260929 17:48:17.183421 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf8650ac64408a809da424f71329eb" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc" is_pre_election: true
I20260929 17:48:17.183584 11782 raft_consensus.cc:2454] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 0.
I20260929 17:48:17.183665 11574 leader_election.cc:302] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, b58cd9b60c444cea92a9ee7c417bf646; no voters:
I20260929 17:48:17.183979 11864 raft_consensus.cc:2781] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:17.184060 11864 raft_consensus.cc:482] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:17.184113 11864 raft_consensus.cc:3037] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.185519 11864 raft_consensus.cc:504] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.185897 11864 leader_election.cc:288] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 election: Requested vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:17.186264 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf8650ac64408a809da424f71329eb" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646"
I20260929 17:48:17.186301 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf8650ac64408a809da424f71329eb" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc"
I20260929 17:48:17.186388 11502 raft_consensus.cc:3037] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.186398 11782 raft_consensus.cc:3037] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.187551 11502 raft_consensus.cc:2454] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 1.
I20260929 17:48:17.187551 11782 raft_consensus.cc:2454] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 1.
I20260929 17:48:17.187844 11574 leader_election.cc:302] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, b58cd9b60c444cea92a9ee7c417bf646; no voters:
I20260929 17:48:17.188000 11864 raft_consensus.cc:2781] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:17.188218 11864 raft_consensus.cc:686] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 LEADER]: Becoming Leader. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Running, Role: LEADER
I20260929 17:48:17.188463 11864 consensus_queue.cc:237] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.189584 11902 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: TxnStatusTablet state changed. Reason: New leader 8598767386d34aaf9fb66a24ab8d5d16. Latest consensus state: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.189697 11902 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:48:17.189958 11904 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:48:17.190110 11904 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:48:17.190771 11345 catalog_manager.cc:5495] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:17.201565 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.198039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39463 (local address 127.0.0.1:44378)
0929 17:48:17.198258 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.198272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.198377 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.198668 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.198671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.198678 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:48:17.198776 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.198781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.199464 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.199467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.200138 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.200148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.200568 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.200572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.200665 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.200706 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.201291 (+ 585us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.201306 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.201366 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.201454 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 17:48:17.201560 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.198118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.3.183.194:39463)
0929 17:48:17.198226 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.198230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.198332 (+ 102us) server_negotiation.cc:407] Connection header received
0929 17:48:17.198516 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.198519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.198571 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.198658 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:17.198865 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.199368 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.200245 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.200478 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.200560 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.200826 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.201204 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.201297 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.201377 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.201430 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:48:17.204537 11782 raft_consensus.cc:1261] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Refusing update from remote peer 8598767386d34aaf9fb66a24ab8d5d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.205013 11902 consensus_queue.cc:1035] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.205109 11502 raft_consensus.cc:1261] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Refusing update from remote peer 8598767386d34aaf9fb66a24ab8d5d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.205592 11902 consensus_queue.cc:1035] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.206702 11863 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: TxnStatusTablet state changed. Reason: New leader 8598767386d34aaf9fb66a24ab8d5d16. Latest consensus state: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.206816 11863 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:48:17.208304 11876 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: TxnStatusTablet state changed. Reason: New leader 8598767386d34aaf9fb66a24ab8d5d16. Latest consensus state: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.208417 11876 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:48:17.213263 11902 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.213359 11902 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:48:17.213876 11872 mvcc.cc:205] Tried to move back new op lower bound from 7334723982144774144 to 7334723982086348800. Current Snapshot: MvccSnapshot[applied={T|T < 7334723982144774144 or (T in {7334723982144774144})}]
I20260929 17:48:17.215586 11870 mvcc.cc:205] Tried to move back new op lower bound from 7334723982144774144 to 7334723982086348800. Current Snapshot: MvccSnapshot[applied={T|T < 7334723982144774144}]
I20260929 17:48:17.215694 11876 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.215811 11876 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P b58cd9b60c444cea92a9ee7c417bf646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:48:17.216833 11909 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.216933 11909 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P 8598767386d34aaf9fb66a24ab8d5d16: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:48:17.218178 11863 tablet_replica.cc:439] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } } }
I20260929 17:48:17.218288 11863 tablet_replica.cc:442] T 0faf8650ac64408a809da424f71329eb P dc4c61130a02403fae4118e7b15ceffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:48:17.223383 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.220009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.3.183.195:46623)
0929 17:48:17.220118 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.220122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.220194 (+ 72us) server_negotiation.cc:407] Connection header received
0929 17:48:17.220333 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.220336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.220392 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.220540 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:17.220750 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.221248 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.222043 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.222262 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.222342 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.222596 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.223010 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.223115 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.223184 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.223245 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 17:48:17.233337 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.230311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:60043 (local address 127.3.183.194:39463)
0929 17:48:17.230421 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.230426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.230588 (+ 162us) server_negotiation.cc:407] Connection header received
0929 17:48:17.230693 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.230696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.230739 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.230795 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:17.231038 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.231514 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.232391 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.232604 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.232684 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.232944 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.233011 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.233066 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.233154 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.233203 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:48:17.233906 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.230230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39463 (local address 127.3.183.195:60043)
0929 17:48:17.230513 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.230525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.230614 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.230809 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.230812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.230819 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:48:17.230937 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.230948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.231604 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.231608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.232287 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.232296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.232693 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.232696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.232787 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.232824 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.233079 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.233085 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.233135 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.233210 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260929 17:48:17.240023 11642 raft_consensus.cc:1261] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Refusing update from remote peer dc4c61130a02403fae4118e7b15ceffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.240082 11502 raft_consensus.cc:1261] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Refusing update from remote peer dc4c61130a02403fae4118e7b15ceffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.240447 11888 consensus_queue.cc:1035] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.240814 11863 consensus_queue.cc:1035] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.262727 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.259111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:38885 (local address 127.3.183.195:46623)
0929 17:48:17.259303 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.259308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.259379 (+ 71us) server_negotiation.cc:407] Connection header received
0929 17:48:17.259516 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.259519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.259564 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.259625 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:17.259878 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.260483 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.261709 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.261922 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.261995 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.262296 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.262389 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.262438 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.262519 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.262560 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260929 17:48:17.262763 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.258998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:46623 (local address 127.3.183.194:38885)
0929 17:48:17.259312 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.259326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.259426 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.259649 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.259651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.259658 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:48:17.259772 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.259801 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.260611 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.260615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.261604 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.261615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.262013 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.262016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.262117 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.262167 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.262445 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.262451 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.262510 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.262597 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260929 17:48:17.282771 11341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:17.282930 11341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:17.285014 11341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Sending DeleteTablet for 3 replicas of tablet 3b59550b04464d35b0c708b309048f33
I20260929 17:48:17.285542 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b59550b04464d35b0c708b309048f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:59896
I20260929 17:48:17.285712 11881 tablet_replica.cc:330] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: stopping tablet replica
I20260929 17:48:17.285813 11881 raft_consensus.cc:2229] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:17.285910 11881 raft_consensus.cc:2258] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.285897 11622 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b59550b04464d35b0c708b309048f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260929 17:48:17.286063 11882 tablet_replica.cc:330] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: stopping tablet replica
I20260929 17:48:17.286183 11882 raft_consensus.cc:2229] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:17.286295 11882 raft_consensus.cc:2258] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.286616 11762 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b59550b04464d35b0c708b309048f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260929 17:48:17.286856 11880 tablet_replica.cc:330] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: stopping tablet replica
I20260929 17:48:17.286952 11880 raft_consensus.cc:2229] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:48:17.286999 11881 ts_tablet_manager.cc:1905] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.287003 11882 ts_tablet_manager.cc:1905] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.287199 11880 raft_consensus.cc:2258] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.289644 11882 ts_tablet_manager.cc:1918] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:48:17.289721 11882 log.cc:1196] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/3b59550b04464d35b0c708b309048f33
I20260929 17:48:17.289707 11881 ts_tablet_manager.cc:1918] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:48:17.289773 11881 log.cc:1196] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/3b59550b04464d35b0c708b309048f33
I20260929 17:48:17.289969 11882 ts_tablet_manager.cc:1939] T 3b59550b04464d35b0c708b309048f33 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting consensus metadata
I20260929 17:48:17.290019 11881 ts_tablet_manager.cc:1939] T 3b59550b04464d35b0c708b309048f33 P b58cd9b60c444cea92a9ee7c417bf646: Deleting consensus metadata
I20260929 17:48:17.290577 11331 catalog_manager.cc:4841] TS 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463): tablet 3b59550b04464d35b0c708b309048f33 (table test-table [id=dd1ab704785b458985a152d1269c93a5]) successfully deleted
I20260929 17:48:17.290730 11880 ts_tablet_manager.cc:1905] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.293236 11880 ts_tablet_manager.cc:1918] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:48:17.293311 11880 log.cc:1196] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/3b59550b04464d35b0c708b309048f33
I20260929 17:48:17.293533 11880 ts_tablet_manager.cc:1939] T 3b59550b04464d35b0c708b309048f33 P dc4c61130a02403fae4118e7b15ceffc: Deleting consensus metadata
I20260929 17:48:17.294159 11330 catalog_manager.cc:4841] TS dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623): tablet 3b59550b04464d35b0c708b309048f33 (table test-table [id=dd1ab704785b458985a152d1269c93a5]) successfully deleted
I20260929 17:48:17.294308 11330 catalog_manager.cc:4841] TS b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781): tablet 3b59550b04464d35b0c708b309048f33 (table test-table [id=dd1ab704785b458985a152d1269c93a5]) successfully deleted
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:48:17.322989 11896 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:17.315308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.3.183.254:39773)
0929 17:48:17.315433 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.315438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.315709 (+ 271us) server_negotiation.cc:407] Connection header received
0929 17:48:17.315875 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.315878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.315933 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.316013 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:17.316899 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.317793 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.318918 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.319282 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.319380 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.322859 (+ 3479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54432: BlockingRecv error: recv got EOF from 127.0.0.1:54432 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM
I20260929 17:48:17.338106 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.328176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.3.183.254:39773)
0929 17:48:17.328290 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.328293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.328524 (+ 231us) server_negotiation.cc:407] Connection header received
0929 17:48:17.328635 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.328638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.328684 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.328760 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:17.329461 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.330236 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.331321 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.331649 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.331746 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.334512 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:17.334516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:17.334519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:17.334555 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:17.336265 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:17.336710 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:17.336714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:17.336716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:17.336814 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:17.337212 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:17.337215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:17.337217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:17.337449 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:17.337579 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.337803 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.337928 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:48:17.341180 11341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:17.341636 11341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:17.346457 11482 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7fd972bba34f738fe55bdbafe5bad9 (DEFAULT_TABLE table=test-table [id=79ac2eef81404b498b9b5af8c67e309b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:17.346633 11622 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7fd972bba34f738fe55bdbafe5bad9 (DEFAULT_TABLE table=test-table [id=79ac2eef81404b498b9b5af8c67e309b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:17.346738 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7fd972bba34f738fe55bdbafe5bad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.346882 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7fd972bba34f738fe55bdbafe5bad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.349129 11856 tablet_bootstrap.cc:492] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: Bootstrap starting.
I20260929 17:48:17.349938 11856 tablet_bootstrap.cc:654] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.350471 11858 tablet_bootstrap.cc:492] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: Bootstrap starting.
I20260929 17:48:17.351357 11858 tablet_bootstrap.cc:654] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.352362 11762 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7fd972bba34f738fe55bdbafe5bad9 (DEFAULT_TABLE table=test-table [id=79ac2eef81404b498b9b5af8c67e309b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:17.352612 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7fd972bba34f738fe55bdbafe5bad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.354796 11857 tablet_bootstrap.cc:492] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: Bootstrap starting.
I20260929 17:48:17.355592 11856 tablet_bootstrap.cc:492] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: No bootstrap required, opened a new log
I20260929 17:48:17.355674 11856 ts_tablet_manager.cc:1397] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 17:48:17.356022 11857 tablet_bootstrap.cc:654] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.356065 11856 raft_consensus.cc:348] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.356686 11858 tablet_bootstrap.cc:492] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: No bootstrap required, opened a new log
I20260929 17:48:17.356763 11858 ts_tablet_manager.cc:1397] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260929 17:48:17.357141 11858 raft_consensus.cc:348] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.357271 11858 raft_consensus.cc:374] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.357299 11858 raft_consensus.cc:729] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.357409 11858 consensus_queue.cc:260] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.357656 11858 ts_tablet_manager.cc:1428] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:48:17.358431 11856 raft_consensus.cc:374] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.358489 11856 raft_consensus.cc:729] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58cd9b60c444cea92a9ee7c417bf646, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.358601 11856 consensus_queue.cc:260] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.358870 11856 ts_tablet_manager.cc:1428] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:48:17.373646 11857 tablet_bootstrap.cc:492] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: No bootstrap required, opened a new log
I20260929 17:48:17.373764 11857 ts_tablet_manager.cc:1397] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20260929 17:48:17.374135 11857 raft_consensus.cc:348] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.374236 11857 raft_consensus.cc:374] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.374264 11857 raft_consensus.cc:729] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4c61130a02403fae4118e7b15ceffc, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.374385 11857 consensus_queue.cc:260] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.374616 11857 ts_tablet_manager.cc:1428] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.429327 11864 raft_consensus.cc:482] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:17.429476 11864 raft_consensus.cc:504] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.429889 11864 leader_election.cc:288] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:17.430267 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7fd972bba34f738fe55bdbafe5bad9" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646" is_pre_election: true
I20260929 17:48:17.430377 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7fd972bba34f738fe55bdbafe5bad9" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc" is_pre_election: true
I20260929 17:48:17.430433 11502 raft_consensus.cc:2454] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 0.
I20260929 17:48:17.430500 11782 raft_consensus.cc:2454] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 0.
I20260929 17:48:17.430825 11574 leader_election.cc:302] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, b58cd9b60c444cea92a9ee7c417bf646; no voters:
I20260929 17:48:17.431001 11864 raft_consensus.cc:2781] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:17.431064 11864 raft_consensus.cc:482] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:17.431102 11864 raft_consensus.cc:3037] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.432135 11864 raft_consensus.cc:504] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.432412 11864 leader_election.cc:288] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 election: Requested vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:17.432749 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7fd972bba34f738fe55bdbafe5bad9" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc"
I20260929 17:48:17.432770 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7fd972bba34f738fe55bdbafe5bad9" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646"
I20260929 17:48:17.432862 11782 raft_consensus.cc:3037] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.432894 11502 raft_consensus.cc:3037] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.433912 11782 raft_consensus.cc:2454] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 1.
I20260929 17:48:17.434108 11502 raft_consensus.cc:2454] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 1.
I20260929 17:48:17.434209 11574 leader_election.cc:302] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, dc4c61130a02403fae4118e7b15ceffc; no voters:
I20260929 17:48:17.434505 11864 raft_consensus.cc:2781] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:17.434599 11864 raft_consensus.cc:686] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 LEADER]: Becoming Leader. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Running, Role: LEADER
I20260929 17:48:17.434733 11864 consensus_queue.cc:237] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.436384 11341 catalog_manager.cc:5495] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:17.450923 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.447292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.3.183.194:39463)
0929 17:48:17.447398 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.447402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.447415 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:17.447457 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.447460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.447504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.447581 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:17.447805 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.448696 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.449941 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.450172 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.450243 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.450520 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.450602 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.450668 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.450755 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.450802 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:48:17.453068 11782 raft_consensus.cc:1261] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Refusing update from remote peer 8598767386d34aaf9fb66a24ab8d5d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.453076 11502 raft_consensus.cc:1261] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Refusing update from remote peer 8598767386d34aaf9fb66a24ab8d5d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.453492 11902 consensus_queue.cc:1035] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.453652 11864 consensus_queue.cc:1035] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.463006 11341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:17.463203 11341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:17.465052 11341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Sending DeleteTablet for 3 replicas of tablet 5f7fd972bba34f738fe55bdbafe5bad9
I20260929 17:48:17.465758 11762 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f7fd972bba34f738fe55bdbafe5bad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260929 17:48:17.465776 11622 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f7fd972bba34f738fe55bdbafe5bad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260929 17:48:17.465936 11882 tablet_replica.cc:330] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: stopping tablet replica
I20260929 17:48:17.465940 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f7fd972bba34f738fe55bdbafe5bad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:59896
I20260929 17:48:17.466037 11882 raft_consensus.cc:2229] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:48:17.466109 11880 tablet_replica.cc:330] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: stopping tablet replica
I20260929 17:48:17.466224 11880 raft_consensus.cc:2229] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:17.466308 11882 raft_consensus.cc:2258] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.466339 11880 raft_consensus.cc:2258] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.466511 11881 tablet_replica.cc:330] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: stopping tablet replica
I20260929 17:48:17.466617 11881 raft_consensus.cc:2229] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:17.466724 11881 raft_consensus.cc:2258] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.467352 11882 ts_tablet_manager.cc:1905] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.467352 11880 ts_tablet_manager.cc:1905] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.468995 11881 ts_tablet_manager.cc:1905] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.469746 11882 ts_tablet_manager.cc:1918] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:17.469746 11880 ts_tablet_manager.cc:1918] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:17.469815 11882 log.cc:1196] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/5f7fd972bba34f738fe55bdbafe5bad9
I20260929 17:48:17.469844 11880 log.cc:1196] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/5f7fd972bba34f738fe55bdbafe5bad9
I20260929 17:48:17.470058 11880 ts_tablet_manager.cc:1939] T 5f7fd972bba34f738fe55bdbafe5bad9 P dc4c61130a02403fae4118e7b15ceffc: Deleting consensus metadata
I20260929 17:48:17.470058 11882 ts_tablet_manager.cc:1939] T 5f7fd972bba34f738fe55bdbafe5bad9 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting consensus metadata
I20260929 17:48:17.470849 11331 catalog_manager.cc:4841] TS 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463): tablet 5f7fd972bba34f738fe55bdbafe5bad9 (table test-table [id=79ac2eef81404b498b9b5af8c67e309b]) successfully deleted
I20260929 17:48:17.471256 11881 ts_tablet_manager.cc:1918] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:17.471324 11881 log.cc:1196] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/5f7fd972bba34f738fe55bdbafe5bad9
I20260929 17:48:17.471342 11341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:48:17.471539 11881 ts_tablet_manager.cc:1939] T 5f7fd972bba34f738fe55bdbafe5bad9 P b58cd9b60c444cea92a9ee7c417bf646: Deleting consensus metadata
W20260929 17:48:17.471729 11341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:17.471849 11330 catalog_manager.cc:4841] TS dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623): tablet 5f7fd972bba34f738fe55bdbafe5bad9 (table test-table [id=79ac2eef81404b498b9b5af8c67e309b]) successfully deleted
I20260929 17:48:17.472117 11330 catalog_manager.cc:4841] TS b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781): tablet 5f7fd972bba34f738fe55bdbafe5bad9 (table test-table [id=79ac2eef81404b498b9b5af8c67e309b]) successfully deleted
I20260929 17:48:17.475744 11622 tablet_service.cc:1467] Processing CreateTablet for tablet fd389506fbf94a20a72653ea8c27d5a3 (DEFAULT_TABLE table=test-table [id=a04073f8adf84ecbb027efa98a9f097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:17.475871 11482 tablet_service.cc:1467] Processing CreateTablet for tablet fd389506fbf94a20a72653ea8c27d5a3 (DEFAULT_TABLE table=test-table [id=a04073f8adf84ecbb027efa98a9f097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:17.476011 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd389506fbf94a20a72653ea8c27d5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.476105 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd389506fbf94a20a72653ea8c27d5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.476082 11762 tablet_service.cc:1467] Processing CreateTablet for tablet fd389506fbf94a20a72653ea8c27d5a3 (DEFAULT_TABLE table=test-table [id=a04073f8adf84ecbb027efa98a9f097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:17.476284 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd389506fbf94a20a72653ea8c27d5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:17.478107 11858 tablet_bootstrap.cc:492] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: Bootstrap starting.
I20260929 17:48:17.478315 11856 tablet_bootstrap.cc:492] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646: Bootstrap starting.
I20260929 17:48:17.478856 11858 tablet_bootstrap.cc:654] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.478981 11856 tablet_bootstrap.cc:654] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.479460 11857 tablet_bootstrap.cc:492] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: Bootstrap starting.
I20260929 17:48:17.479948 11858 tablet_bootstrap.cc:492] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: No bootstrap required, opened a new log
I20260929 17:48:17.480026 11858 ts_tablet_manager.cc:1397] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 17:48:17.480150 11857 tablet_bootstrap.cc:654] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:17.480369 11858 raft_consensus.cc:348] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.480475 11858 raft_consensus.cc:374] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.480504 11858 raft_consensus.cc:729] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.480623 11858 consensus_queue.cc:260] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.480862 11858 ts_tablet_manager.cc:1428] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.481432 11856 tablet_bootstrap.cc:492] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646: No bootstrap required, opened a new log
I20260929 17:48:17.481526 11856 ts_tablet_manager.cc:1397] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:48:17.481897 11857 tablet_bootstrap.cc:492] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: No bootstrap required, opened a new log
I20260929 17:48:17.481899 11856 raft_consensus.cc:348] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.481984 11857 ts_tablet_manager.cc:1397] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:48:17.481994 11856 raft_consensus.cc:374] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.482024 11856 raft_consensus.cc:729] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58cd9b60c444cea92a9ee7c417bf646, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.482146 11856 consensus_queue.cc:260] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.482380 11856 ts_tablet_manager.cc:1428] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.482365 11857 raft_consensus.cc:348] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.482460 11857 raft_consensus.cc:374] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:17.482491 11857 raft_consensus.cc:729] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4c61130a02403fae4118e7b15ceffc, State: Initialized, Role: FOLLOWER
I20260929 17:48:17.482592 11857 consensus_queue.cc:260] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.482833 11857 ts_tablet_manager.cc:1428] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:48:17.524945 11864 raft_consensus.cc:482] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:17.525080 11864 raft_consensus.cc:504] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.525471 11864 leader_election.cc:288] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:17.525954 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd389506fbf94a20a72653ea8c27d5a3" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc" is_pre_election: true
I20260929 17:48:17.525974 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd389506fbf94a20a72653ea8c27d5a3" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646" is_pre_election: true
I20260929 17:48:17.526113 11502 raft_consensus.cc:2454] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 0.
I20260929 17:48:17.526114 11782 raft_consensus.cc:2454] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 0.
I20260929 17:48:17.526402 11574 leader_election.cc:302] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, b58cd9b60c444cea92a9ee7c417bf646; no voters:
I20260929 17:48:17.526602 11864 raft_consensus.cc:2781] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:17.526669 11864 raft_consensus.cc:482] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:17.526705 11864 raft_consensus.cc:3037] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.527891 11864 raft_consensus.cc:504] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.528210 11864 leader_election.cc:288] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 election: Requested vote from peers b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781), dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623)
I20260929 17:48:17.528621 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd389506fbf94a20a72653ea8c27d5a3" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4c61130a02403fae4118e7b15ceffc"
I20260929 17:48:17.528725 11782 raft_consensus.cc:3037] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.529633 11782 raft_consensus.cc:2454] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 1.
I20260929 17:48:17.529872 11502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd389506fbf94a20a72653ea8c27d5a3" candidate_uuid: "8598767386d34aaf9fb66a24ab8d5d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58cd9b60c444cea92a9ee7c417bf646"
I20260929 17:48:17.529927 11574 leader_election.cc:302] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8598767386d34aaf9fb66a24ab8d5d16, dc4c61130a02403fae4118e7b15ceffc; no voters:
I20260929 17:48:17.530009 11502 raft_consensus.cc:3037] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:17.530150 11864 raft_consensus.cc:2781] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:17.530229 11864 raft_consensus.cc:686] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 LEADER]: Becoming Leader. State: Replica: 8598767386d34aaf9fb66a24ab8d5d16, State: Running, Role: LEADER
I20260929 17:48:17.530349 11864 consensus_queue.cc:237] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } }
I20260929 17:48:17.531117 11502 raft_consensus.cc:2454] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8598767386d34aaf9fb66a24ab8d5d16 in term 1.
I20260929 17:48:17.532015 11345 catalog_manager.cc:5495] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "8598767386d34aaf9fb66a24ab8d5d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8598767386d34aaf9fb66a24ab8d5d16" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:17.558754 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.555944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:59995 (local address 127.3.183.194:39463)
0929 17:48:17.556031 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:17.556035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:17.556174 (+ 139us) server_negotiation.cc:407] Connection header received
0929 17:48:17.556198 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:17.556200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:17.556228 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:17.556282 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:17.556553 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.557126 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.557887 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.558103 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.558182 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.558371 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.558441 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.558493 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:17.558569 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 17:48:17.558619 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":36}
I20260929 17:48:17.558754 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.555905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39463 (local address 127.3.183.194:59995)
0929 17:48:17.556016 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.556031 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:17.556116 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:17.556294 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:17.556297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:17.556303 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:48:17.556450 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.556455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.557227 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.557230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.557791 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:17.557798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:17.558186 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:17.558189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:17.558261 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:17.558284 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:48:17.558502 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:48:17.558507 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:48:17.558562 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 17:48:17.558642 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":22}
I20260929 17:48:17.563954 11782 raft_consensus.cc:1261] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Refusing update from remote peer 8598767386d34aaf9fb66a24ab8d5d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.564432 11864 consensus_queue.cc:1035] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4c61130a02403fae4118e7b15ceffc" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.564543 11502 raft_consensus.cc:1261] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Refusing update from remote peer 8598767386d34aaf9fb66a24ab8d5d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:17.564965 11864 consensus_queue.cc:1035] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58cd9b60c444cea92a9ee7c417bf646" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:17.594784 11341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:17.594949 11341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:17.596838 11341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f89ba88fc304dad9b61f07bea5a2312: Sending DeleteTablet for 3 replicas of tablet fd389506fbf94a20a72653ea8c27d5a3
I20260929 17:48:17.597349 11622 tablet_service.cc:1514] Processing DeleteTablet for tablet fd389506fbf94a20a72653ea8c27d5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260929 17:48:17.597483 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet fd389506fbf94a20a72653ea8c27d5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:59896
I20260929 17:48:17.597537 11882 tablet_replica.cc:330] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: stopping tablet replica
I20260929 17:48:17.597648 11881 tablet_replica.cc:330] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646: stopping tablet replica
I20260929 17:48:17.597653 11882 raft_consensus.cc:2229] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:48:17.597681 11762 tablet_service.cc:1514] Processing DeleteTablet for tablet fd389506fbf94a20a72653ea8c27d5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:17 UTC) from {username='oryx', principal='oryx/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260929 17:48:17.597740 11881 raft_consensus.cc:2229] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:17.597832 11881 raft_consensus.cc:2258] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.597829 11880 tablet_replica.cc:330] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: stopping tablet replica
I20260929 17:48:17.597836 11882 raft_consensus.cc:2258] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.597932 11880 raft_consensus.cc:2229] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:17.598021 11880 raft_consensus.cc:2258] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:17.598603 11882 ts_tablet_manager.cc:1905] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.598615 11881 ts_tablet_manager.cc:1905] T fd389506fbf94a20a72653ea8c27d5a3 P b58cd9b60c444cea92a9ee7c417bf646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.599116 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 11417
I20260929 17:48:17.600996 11882 ts_tablet_manager.cc:1918] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:48:17.601058 11882 log.cc:1196] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/fd389506fbf94a20a72653ea8c27d5a3
I20260929 17:48:17.601274 11882 ts_tablet_manager.cc:1939] T fd389506fbf94a20a72653ea8c27d5a3 P 8598767386d34aaf9fb66a24ab8d5d16: Deleting consensus metadata
I20260929 17:48:17.601862 11331 catalog_manager.cc:4841] TS 8598767386d34aaf9fb66a24ab8d5d16 (127.3.183.194:39463): tablet fd389506fbf94a20a72653ea8c27d5a3 (table test-table [id=a04073f8adf84ecbb027efa98a9f097b]) successfully deleted
I20260929 17:48:17.604389 11880 ts_tablet_manager.cc:1905] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:17.606693 11880 ts_tablet_manager.cc:1918] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:48:17.606771 11880 log.cc:1196] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/fd389506fbf94a20a72653ea8c27d5a3
I20260929 17:48:17.607055 11880 ts_tablet_manager.cc:1939] T fd389506fbf94a20a72653ea8c27d5a3 P dc4c61130a02403fae4118e7b15ceffc: Deleting consensus metadata
I20260929 17:48:17.607734 11330 catalog_manager.cc:4841] TS dc4c61130a02403fae4118e7b15ceffc (127.3.183.195:46623): tablet fd389506fbf94a20a72653ea8c27d5a3 (table test-table [id=a04073f8adf84ecbb027efa98a9f097b]) successfully deleted
W20260929 17:48:17.608762 11330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:39781 (error 108)
I20260929 17:48:17.608886 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 11557
I20260929 17:48:17.610327 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:17.609946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39781 (local address 127.0.0.1:59924)
0929 17:48:17.610183 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:17.610248 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
W20260929 17:48:17.610437 11330 catalog_manager.cc:4584] TS b58cd9b60c444cea92a9ee7c417bf646 (127.3.183.193:39781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd389506fbf94a20a72653ea8c27d5a3: Network error: Client connection negotiation failed: client connection to 127.3.183.193:39781: connect: Connection refused (error 111)
I20260929 17:48:17.615686 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 11697
I20260929 17:48:17.620940 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 11313
2026-09-29T17:48:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3179 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11950](info): set up 2 sockets
Sep 29 17:48:17 dist-test-slave-wbpt krb5kdc[11950](info): commencing operation
krb5kdc: starting...
W20260929 17:48:18.584408 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:18 dist-test-slave-wbpt krb5kdc[11950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:48:18.707491 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:42663
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:42663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:18.815016 11963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:18.815310 11963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:18.815366 11963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:18.819353 11963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:18.819411 11963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:18.819442 11963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:18.819464 11963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:18.819485 11963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:18.824654 11963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:42663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:42663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:18.825646 11963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:18.826594 11963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:18.834793 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:18.834784 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:18.834842 11963 server_base.cc:1034] running on GCE node
W20260929 17:48:18.834789 11971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:18.835541 11963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:18.836634 11963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:18.837800 11963 hybrid_clock.cc:648] HybridClock initialized: now 1790704098837778 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T17:48:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1225 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:18 dist-test-slave-wbpt krb5kdc[11976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:18 dist-test-slave-wbpt krb5kdc[11976](info): set up 2 sockets
Sep 29 17:48:18 dist-test-slave-wbpt krb5kdc[11976](info): commencing operation
krb5kdc: starting...
W20260929 17:48:19.808902 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:19 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:48:19.927618 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46745
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:46745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:20.037350 11989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:20.037673 11989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:20.037735 11989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:20.042261 11989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:20.042372 11989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:20.042402 11989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:20.042425 11989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:20.042445 11989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:20.048399 11989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:46745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:20.049558 11989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:20.050647 11989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:20.059382 11994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:20.059374 11995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:20.059468 11989 server_base.cc:1034] running on GCE node
W20260929 17:48:20.059370 11997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:20.060185 11989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:20.061432 11989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:20.062614 11989 hybrid_clock.cc:648] HybridClock initialized: now 1790704100062584 us; error 43 us; skew 500 ppm
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:20.071646 11989 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:48:20.072743 11989 webserver.cc:460] Webserver started at http://127.3.183.254:42473/ using document root <none> and password file <none>
I20260929 17:48:20.073324 11989 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:20.073400 11989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:20.073597 11989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:20.075752 11989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "d8fe5a8dfee943c5b8ca311a0773fe66"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "49426a43347b6c02a1def9a072f35858"
server_key_iv: "686343cd189c69ef04b7ba531e7013f7"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.076359 11989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "d8fe5a8dfee943c5b8ca311a0773fe66"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "49426a43347b6c02a1def9a072f35858"
server_key_iv: "686343cd189c69ef04b7ba531e7013f7"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.080317 11989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:48:20.082979 12004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.084434 11989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:48:20.084582 11989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "d8fe5a8dfee943c5b8ca311a0773fe66"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "49426a43347b6c02a1def9a072f35858"
server_key_iv: "686343cd189c69ef04b7ba531e7013f7"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.084700 11989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:20.109112 11989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:20.112483 11989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:20.112674 11989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:20.119376 11989 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:46745
I20260929 17:48:20.119396 12056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:46745 every 8 connection(s)
I20260929 17:48:20.120474 11989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:20.123253 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:20.124091 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 11989
I20260929 17:48:20.124203 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:20.124475 3807 external_mini_cluster.cc:1428] Setting key 636840691e5146288bf4d38a58d97272
I20260929 17:48:20.129210 12057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Bootstrap starting.
I20260929 17:48:20.131400 12057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:20.132319 12057 log.cc:824] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:20.134578 12057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: No bootstrap required, opened a new log
I20260929 17:48:20.138399 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:20.125900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.3.183.254:46745)
0929 17:48:20.126351 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:20.126367 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:20.126402 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:48:20.127061 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:20.127080 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:20.127414 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:20.127727 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:20.128541 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.129546 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.130393 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.130872 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.131017 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:20.133912 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:20.133917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:20.133931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:20.133957 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:20.135644 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.136129 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.136133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.136138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.136244 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.136533 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.136535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.136536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.136923 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:20.137064 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:20.137345 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 17:48:20.137634 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260929 17:48:20.138484 12057 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:20.138761 12057 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:20.138815 12057 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fe5a8dfee943c5b8ca311a0773fe66, State: Initialized, Role: FOLLOWER
I20260929 17:48:20.139249 12057 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:20.139394 12057 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:20.139464 12057 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:20.139562 12057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:20.140458 12057 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:20.140803 12057 leader_election.cc:302] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8fe5a8dfee943c5b8ca311a0773fe66; no voters:
I20260929 17:48:20.141057 12057 leader_election.cc:288] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:20.141166 12062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:20.141372 12062 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 LEADER]: Becoming Leader. State: Replica: d8fe5a8dfee943c5b8ca311a0773fe66, State: Running, Role: LEADER
I20260929 17:48:20.141682 12062 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:20.142151 12057 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:20.143179 12064 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8fe5a8dfee943c5b8ca311a0773fe66. Latest consensus state: current_term: 1 leader_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } } }
I20260929 17:48:20.143301 12064 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:20.143689 12071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:20.143652 12063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } } }
I20260929 17:48:20.143821 12063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:20.147028 12071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:20.151850 12071 catalog_manager.cc:1347] Generated new cluster ID: 260846e057d34c3f886092654321afc3
I20260929 17:48:20.151914 12071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:20.165002 12071 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:20.166880 12071 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:20.176784 12071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Generated new TSK 0
I20260929 17:48:20.177672 12071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:48:20.208976 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46745
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:20.327306 12083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:20.327592 12083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:20.327641 12083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:20.331677 12083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:20.331743 12083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:20.331835 12083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:20.337268 12083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:20.338351 12083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:20.339375 12083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:48:20.348798 12083 server_base.cc:1034] running on GCE node
W20260929 17:48:20.348759 12089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:20.348778 12088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:20.348733 12091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:20.349318 12083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:20.349946 12083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:20.351110 12083 hybrid_clock.cc:648] HybridClock initialized: now 1790704100351089 us; error 32 us; skew 500 ppm
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:20.353977 12083 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:48:20.354863 12083 webserver.cc:460] Webserver started at http://127.3.183.193:35235/ using document root <none> and password file <none>
I20260929 17:48:20.355383 12083 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:20.355448 12083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:20.355664 12083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:20.357728 12083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "64ece2aa0bd34c759c13904b1f84a131"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "6ff4ddb158937e8f25cb9b424c8f6831"
server_key_iv: "1d91b4c27cb5180ce93000dc4f3fe96c"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.358255 12083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "64ece2aa0bd34c759c13904b1f84a131"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "6ff4ddb158937e8f25cb9b424c8f6831"
server_key_iv: "1d91b4c27cb5180ce93000dc4f3fe96c"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.361909 12083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:48:20.364378 12098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.365631 12083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:20.365734 12083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "64ece2aa0bd34c759c13904b1f84a131"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "6ff4ddb158937e8f25cb9b424c8f6831"
server_key_iv: "1d91b4c27cb5180ce93000dc4f3fe96c"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.365824 12083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:20.389487 12083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:20.392351 12083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:20.392532 12083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:20.393105 12083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:20.393901 12083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:20.393950 12083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.394003 12083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:20.394027 12083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.402385 12083 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:37167
I20260929 17:48:20.402415 12211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:37167 every 8 connection(s)
I20260929 17:48:20.403225 12083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:20.405652 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 12083
I20260929 17:48:20.405773 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:20.406090 3807 external_mini_cluster.cc:1428] Setting key 45def79b72b954a50fe1b16866a5421b
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:48:20.416147 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:20.404969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:33207 (local address 127.3.183.254:46745)
0929 17:48:20.405084 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:20.405088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:20.405971 (+ 883us) server_negotiation.cc:407] Connection header received
0929 17:48:20.407314 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:20.407317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:20.407371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:20.407441 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:20.408650 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.409180 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.410293 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.410522 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.410619 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:20.412886 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:20.412889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:20.412891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:20.412914 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:20.414226 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.414665 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.414668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.414670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.414743 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.415031 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.415034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.415036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.415230 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:20.415314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:20.415805 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 17:48:20.415938 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:48:20.416941 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:20.405310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:33207)
0929 17:48:20.405822 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:20.405877 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:20.407113 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:20.407622 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:20.407630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:20.408031 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:20.408506 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:20.408520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.409295 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.409298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:20.410175 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:20.410183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.410664 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.410668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:20.410826 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:20.411359 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:20.411375 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:20.412737 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:20.414319 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:20.414325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:20.414340 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:20.414575 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:20.414821 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:20.414824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:20.414826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:20.414955 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:20.415323 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:20.415329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:20.415561 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 17:48:20.415731 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 17:48:20.416070 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:48:20.418136 12212 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46745
I20260929 17:48:20.418399 12212 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:20.418943 12212 heartbeater.cc:511] Master 127.3.183.254:46745 requested a full tablet report, sending...
I20260929 17:48:20.420389 12021 ts_manager.cc:194] Registered new tserver with Master: 64ece2aa0bd34c759c13904b1f84a131 (127.3.183.193:37167)
I20260929 17:48:20.421487 12021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:33207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:48:20.432384 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46745
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:20.542198 12217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:20.542475 12217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:20.542536 12217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:20.546875 12217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:20.546945 12217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:20.547030 12217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:20.552385 12217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:20.553515 12217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:20.554543 12217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:20.564189 12222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:20.564200 12225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:20.564214 12223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:20.564270 12217 server_base.cc:1034] running on GCE node
I20260929 17:48:20.564862 12217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:20.565469 12217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:20.566637 12217 hybrid_clock.cc:648] HybridClock initialized: now 1790704100566618 us; error 33 us; skew 500 ppm
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:20.569329 12217 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:48:20.570192 12217 webserver.cc:460] Webserver started at http://127.3.183.194:40147/ using document root <none> and password file <none>
I20260929 17:48:20.570720 12217 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:20.570780 12217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:20.570940 12217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:20.573058 12217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "5abfbb6982894fa3aad2695fc6543a09"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "084f2d698feb77ea68c8f091f6e209c1"
server_key_iv: "277f685429a38f2c3180fdc17f342928"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.573702 12217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "5abfbb6982894fa3aad2695fc6543a09"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "084f2d698feb77ea68c8f091f6e209c1"
server_key_iv: "277f685429a38f2c3180fdc17f342928"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.577612 12217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:48:20.580083 12232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.581413 12217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:48:20.581561 12217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "5abfbb6982894fa3aad2695fc6543a09"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "084f2d698feb77ea68c8f091f6e209c1"
server_key_iv: "277f685429a38f2c3180fdc17f342928"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.581674 12217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:20.602591 12217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:20.605695 12217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:20.605863 12217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:20.606451 12217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:20.607250 12217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:20.607296 12217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.607338 12217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:20.607367 12217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.615909 12217 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:35889
I20260929 17:48:20.615942 12345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:35889 every 8 connection(s)
I20260929 17:48:20.616787 12217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:20.618595 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 12217
I20260929 17:48:20.618727 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:20.619026 3807 external_mini_cluster.cc:1428] Setting key 22650743a5c15dc042e2dabbdcc823eb
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:48:20.630357 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:20.618629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:39377 (local address 127.3.183.254:46745)
0929 17:48:20.618837 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:20.618842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:20.619660 (+ 818us) server_negotiation.cc:407] Connection header received
0929 17:48:20.621044 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:20.621048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:20.621099 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:20.621182 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:20.622429 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.622961 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.624166 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.624384 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.624476 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:20.627063 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:20.627068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:20.627071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:20.627099 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:20.628350 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.628892 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.628895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.628896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.628994 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.629256 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.629258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.629259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.629424 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:20.629504 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:20.630101 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 17:48:20.630227 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 17:48:20.631372 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:20.619021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:39377)
0929 17:48:20.619509 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:20.619563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:20.620831 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:20.621372 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:20.621379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:20.621778 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:20.622268 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:20.622282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.623106 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.623109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:20.624045 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:20.624055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.624529 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.624533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:20.624691 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:20.625281 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:20.625297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:20.626888 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:20.628493 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:20.628500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:20.628517 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:20.628816 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:20.629061 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:20.629063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:20.629064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:20.629192 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:20.629500 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:20.629505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:20.629819 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 17:48:20.630020 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 17:48:20.630324 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":129,"threads_started":1}
I20260929 17:48:20.632534 12346 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46745
I20260929 17:48:20.632763 12346 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:20.633277 12346 heartbeater.cc:511] Master 127.3.183.254:46745 requested a full tablet report, sending...
I20260929 17:48:20.634316 12021 ts_manager.cc:194] Registered new tserver with Master: 5abfbb6982894fa3aad2695fc6543a09 (127.3.183.194:35889)
I20260929 17:48:20.634876 12021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:39377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:48:20.648756 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46745
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:20.760350 12351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:20.760643 12351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:20.760689 12351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:20.764853 12351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:20.764922 12351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:20.764984 12351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:20.770519 12351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:20.771698 12351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:20.772785 12351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:20.782907 12357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:20.783003 12351 server_base.cc:1034] running on GCE node
W20260929 17:48:20.782907 12359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:20.782912 12356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:20.783587 12351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:20.784297 12351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:20.785523 12351 hybrid_clock.cc:648] HybridClock initialized: now 1790704100785505 us; error 35 us; skew 500 ppm
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:20.788350 12351 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:48:20.789310 12351 webserver.cc:460] Webserver started at http://127.3.183.195:38213/ using document root <none> and password file <none>
I20260929 17:48:20.789877 12351 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:20.789939 12351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:20.790120 12351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:20.792416 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "4f565388728443769807efd39f1be973"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "9561b5c4b48587ac1cd4d59c13179bd6"
server_key_iv: "bb0df8bb5a4723c44d4b0fa35eae9250"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.792954 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "4f565388728443769807efd39f1be973"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "9561b5c4b48587ac1cd4d59c13179bd6"
server_key_iv: "bb0df8bb5a4723c44d4b0fa35eae9250"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.796849 12351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:48:20.799476 12366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.800882 12351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:48:20.800999 12351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "4f565388728443769807efd39f1be973"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "9561b5c4b48587ac1cd4d59c13179bd6"
server_key_iv: "bb0df8bb5a4723c44d4b0fa35eae9250"
server_key_version: "encryptionkey@0"
I20260929 17:48:20.801113 12351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:20.825871 12351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:20.828884 12351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:20.829062 12351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:20.829638 12351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:20.830442 12351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:20.830488 12351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.830528 12351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:20.830557 12351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:20.838878 12351 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:33519
I20260929 17:48:20.838912 12479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:33519 every 8 connection(s)
I20260929 17:48:20.839758 12351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:20.845532 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 12351
I20260929 17:48:20.845675 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:20.845969 3807 external_mini_cluster.cc:1428] Setting key bf4b9fee9eafad8636feffb6393db1fc
Sep 29 17:48:20 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:20.853238 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:20.841535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:54855 (local address 127.3.183.254:46745)
0929 17:48:20.841665 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:20.841669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:20.842715 (+ 1046us) server_negotiation.cc:407] Connection header received
0929 17:48:20.844145 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:20.844148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:20.844198 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:20.844269 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:20.845604 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.846328 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.847623 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.847893 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.848003 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:20.850274 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:20.850278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:20.850280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:20.850305 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:20.851560 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.852006 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.852009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.852011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.852089 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:20.852347 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:20.852349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:20.852350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:20.852544 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:20.852615 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:20.853026 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 17:48:20.853118 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:48:20.854128 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:20.841913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:54855)
0929 17:48:20.842544 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:20.842608 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:20.843923 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:20.844499 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:20.844506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:20.844891 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:20.845413 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:20.845428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.846456 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.846460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:20.847509 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:20.847519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:20.848037 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:20.848042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:20.848225 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:20.848727 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:20.848742 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:20.850125 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:20.851641 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:20.851646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:20.851658 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:20.851910 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:20.852165 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:20.852166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:20.852168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:20.852287 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:20.852614 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:20.852619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:20.852842 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 17:48:20.852989 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 17:48:20.853253 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
I20260929 17:48:20.855458 12480 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46745
I20260929 17:48:20.855686 12480 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:20.856194 12480 heartbeater.cc:511] Master 127.3.183.254:46745 requested a full tablet report, sending...
I20260929 17:48:20.857209 12021 ts_manager.cc:194] Registered new tserver with Master: 4f565388728443769807efd39f1be973 (127.3.183.195:33519)
I20260929 17:48:20.857770 12021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:54855
I20260929 17:48:20.859886 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab.
Entry for principal oryx/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab.
WARNING: no policy specified for oryx/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab.
Entry for principal oryx/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab.
WARNING: no policy specified for oryx/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab.
Entry for principal oryx/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab.
I20260929 17:48:20.934751 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 12083
I20260929 17:48:20.939522 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 12217
I20260929 17:48:20.944120 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 12351
I20260929 17:48:20.948446 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 11989
I20260929 17:48:20.953629 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46745
--webserver_interface=127.3.183.254
--webserver_port=42473
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:46745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:21.063608 12489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:21.063906 12489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:21.063956 12489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:21.068029 12489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:21.068094 12489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:21.068112 12489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:21.068128 12489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:21.068142 12489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:21.073446 12489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:46745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=42473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:21.074498 12489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:21.075497 12489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:48:21.083879 12489 server_base.cc:1034] running on GCE node
W20260929 17:48:21.083843 12495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:21.083845 12497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:21.083904 12494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:21.084692 12489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:21.085789 12489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:21.086938 12489 hybrid_clock.cc:648] HybridClock initialized: now 1790704101086915 us; error 35 us; skew 500 ppm
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704101, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:21.089555 12489 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:48:21.090401 12489 webserver.cc:460] Webserver started at http://127.3.183.254:42473/ using document root <none> and password file <none>
I20260929 17:48:21.090922 12489 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:21.091008 12489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:21.094863 12489 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:21.097296 12504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.098627 12489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:48:21.098762 12489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "d8fe5a8dfee943c5b8ca311a0773fe66"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "49426a43347b6c02a1def9a072f35858"
server_key_iv: "686343cd189c69ef04b7ba531e7013f7"
server_key_version: "encryptionkey@0"
I20260929 17:48:21.099336 12489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:21.119186 12489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:21.122309 12489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:21.122506 12489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:21.129495 12489 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:46745
I20260929 17:48:21.129585 12556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:46745 every 8 connection(s)
I20260929 17:48:21.130611 12489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:21.135113 12557 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:48:21.136969 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Bootstrap starting.
I20260929 17:48:21.139971 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 12489
I20260929 17:48:21.140388 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:37167
--local_ip_for_outbound_sockets=127.3.183.193
--tserver_master_addrs=127.3.183.254:46745
--webserver_port=35235
--webserver_interface=127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:48:21.154172 12557 log.cc:824] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:21.161186 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:48:21.161557 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Bootstrap complete.
I20260929 17:48:21.165891 12557 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:21.166303 12557 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fe5a8dfee943c5b8ca311a0773fe66, State: Initialized, Role: FOLLOWER
I20260929 17:48:21.166776 12557 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:21.166919 12557 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:21.166987 12557 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:21.167093 12557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:48:21.169071 12557 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:21.169379 12557 leader_election.cc:302] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8fe5a8dfee943c5b8ca311a0773fe66; no voters:
I20260929 17:48:21.169669 12557 leader_election.cc:288] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:48:21.169771 12561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:48:21.169955 12561 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [term 2 LEADER]: Becoming Leader. State: Replica: d8fe5a8dfee943c5b8ca311a0773fe66, State: Running, Role: LEADER
I20260929 17:48:21.170306 12561 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } }
I20260929 17:48:21.170909 12557 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:21.171756 12562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } } }
I20260929 17:48:21.171902 12562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:21.171762 12563 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8fe5a8dfee943c5b8ca311a0773fe66. Latest consensus state: current_term: 2 leader_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe5a8dfee943c5b8ca311a0773fe66" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46745 } } }
I20260929 17:48:21.172211 12563 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:21.172957 12570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:21.175566 12570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:21.176124 12570 catalog_manager.cc:1259] Loaded cluster ID: 260846e057d34c3f886092654321afc3
I20260929 17:48:21.176173 12570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:21.177842 12570 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:21.178356 12570 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8fe5a8dfee943c5b8ca311a0773fe66: Loaded TSK: 0
I20260929 17:48:21.179251 12570 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:48:21.264660 12559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:21.264940 12559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:21.265013 12559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:21.269202 12559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:21.269273 12559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:21.269354 12559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:21.274941 12559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.193.keytab
--principal=oryx/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:37167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=35235
--tserver_master_addrs=127.3.183.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:21.276088 12559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:21.277117 12559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:21.287067 12586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:21.287068 12583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:21.287202 12559 server_base.cc:1034] running on GCE node
W20260929 17:48:21.287083 12584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:21.288058 12559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:21.288712 12559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:21.289876 12559 hybrid_clock.cc:648] HybridClock initialized: now 1790704101289850 us; error 56 us; skew 500 ppm
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704101, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:21.292673 12559 init.cc:377] Logged in from keytab as oryx/127.3.183.193@KRBTEST.COM (short username oryx)
I20260929 17:48:21.293771 12559 webserver.cc:460] Webserver started at http://127.3.183.193:35235/ using document root <none> and password file <none>
I20260929 17:48:21.294485 12559 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:21.294596 12559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:21.299947 12559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:48:21.302542 12593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.304090 12559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:48:21.304199 12559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "64ece2aa0bd34c759c13904b1f84a131"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "6ff4ddb158937e8f25cb9b424c8f6831"
server_key_iv: "1d91b4c27cb5180ce93000dc4f3fe96c"
server_key_version: "encryptionkey@0"
I20260929 17:48:21.304853 12559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:21.333639 12559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:21.336876 12559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:21.337033 12559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:21.337621 12559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:21.338416 12559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:21.338465 12559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.338506 12559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:21.338521 12559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.346946 12559 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:37167
I20260929 17:48:21.347004 12706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:37167 every 8 connection(s)
I20260929 17:48:21.347999 12559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:21.357707 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 12559
I20260929 17:48:21.358219 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:35889
--local_ip_for_outbound_sockets=127.3.183.194
--tserver_master_addrs=127.3.183.254:46745
--webserver_port=40147
--webserver_interface=127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:21.362679 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.350241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:40529)
0929 17:48:21.350813 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.350868 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.352184 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.353505 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.353513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.353924 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.354418 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.354433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.356175 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.356178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.357133 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.357144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.358067 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.358071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.358281 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.358957 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.358978 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.361803 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
W20260929 17:48:21.363119 12709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.350147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:40529 (local address 127.3.183.254:46745)
0929 17:48:21.350649 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.350671 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.351057 (+ 386us) server_negotiation.cc:407] Connection header received
0929 17:48:21.352669 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.352682 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.353045 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.353361 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.354633 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.356009 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.357286 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.357886 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.358033 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.362140 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:40529: BlockingRecv error: recv got EOF from 127.3.183.193:40529 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":167,"threads_started":1}
W20260929 17:48:21.363161 12707 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:46745 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.183.254:46745: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:21.368969 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.363610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:52087)
0929 17:48:21.363715 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.363728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.363830 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.364198 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.364200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.364328 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.364421 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.364426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.365191 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.365196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.366067 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.366079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.366486 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.366490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.366589 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.367017 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.367032 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.368864 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 17:48:21.369170 12709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.363677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:52087 (local address 127.3.183.254:46745)
0929 17:48:21.363973 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.363977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.363993 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:48:21.364038 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.364041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.364093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.364186 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.364538 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.365062 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.366178 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.366385 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.366458 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.369048 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:52087: BlockingRecv error: recv got EOF from 127.3.183.193:52087 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":82}
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:21.374137 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.369656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:57671)
0929 17:48:21.369758 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.369773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.369847 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.370054 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.370056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.370201 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.370278 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.370284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.371013 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.371017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.371531 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.371538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.372007 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.372011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.372087 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.372505 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.372517 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.374004 (+ 1487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 17:48:21.374231 12709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.369718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:57671 (local address 127.3.183.254:46745)
0929 17:48:21.369814 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.369818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.369829 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:48:21.369926 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.369929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.369968 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.370028 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.370380 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.370920 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.371637 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.371905 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.371986 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.374130 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:57671: BlockingRecv error: recv got EOF from 127.3.183.193:57671 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 17:48:21.374846 12707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:48:21.472160 12711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:21.472473 12711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:21.472548 12711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:21.476725 12711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:21.476799 12711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:21.476882 12711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:21.482412 12711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.194.keytab
--principal=oryx/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:35889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=40147
--tserver_master_addrs=127.3.183.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:21.483502 12711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:21.484522 12711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:21.493928 12719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:21.493932 12716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:21.494020 12711 server_base.cc:1034] running on GCE node
W20260929 17:48:21.493968 12717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:21.494638 12711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:21.495273 12711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:21.496431 12711 hybrid_clock.cc:648] HybridClock initialized: now 1790704101496397 us; error 47 us; skew 500 ppm
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704101, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:21.499163 12711 init.cc:377] Logged in from keytab as oryx/127.3.183.194@KRBTEST.COM (short username oryx)
I20260929 17:48:21.500097 12711 webserver.cc:460] Webserver started at http://127.3.183.194:40147/ using document root <none> and password file <none>
I20260929 17:48:21.500648 12711 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:21.500725 12711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:21.504648 12711 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:21.506690 12726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.508067 12711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:21.508176 12711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "5abfbb6982894fa3aad2695fc6543a09"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "084f2d698feb77ea68c8f091f6e209c1"
server_key_iv: "277f685429a38f2c3180fdc17f342928"
server_key_version: "encryptionkey@0"
I20260929 17:48:21.508734 12711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:21.532922 12711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:21.535878 12711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:21.536054 12711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:21.536664 12711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:21.537488 12711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:21.537535 12711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.537586 12711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:21.537611 12711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.546602 12711 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:35889
I20260929 17:48:21.546629 12839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:35889 every 8 connection(s)
I20260929 17:48:21.547628 12711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:21.555390 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 12711
I20260929 17:48:21.555920 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:33519
--local_ip_for_outbound_sockets=127.3.183.195
--tserver_master_addrs=127.3.183.254:46745
--webserver_port=38213
--webserver_interface=127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:48:21.559801 12709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.549515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:45791 (local address 127.3.183.254:46745)
0929 17:48:21.549653 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.549657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.550482 (+ 825us) server_negotiation.cc:407] Connection header received
0929 17:48:21.551846 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.551850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.551920 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.552010 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.553270 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.553810 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.554988 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.555284 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.555419 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.559668 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:45791: BlockingRecv error: recv got EOF from 127.3.183.194:45791 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:48:21.560636 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.549846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:45791)
0929 17:48:21.550320 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.550364 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.551579 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.552201 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.552209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.552596 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.553111 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.553126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.553948 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.553951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.554840 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.554851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.555477 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.555481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.555674 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.556331 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.556356 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.559524 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
W20260929 17:48:21.561141 12840 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:46745 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.183.254:46745: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:21.567073 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.561559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:55815)
0929 17:48:21.561728 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.561741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.561812 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.562155 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.562157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.562289 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.562397 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.562403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.563394 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.563397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.563998 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.564010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.564568 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.564571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.564642 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.565117 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.565129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.566969 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 17:48:21.567256 12709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.561669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:55815 (local address 127.3.183.254:46745)
0929 17:48:21.561849 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.561854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.561870 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:48:21.561946 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.561949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.562017 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.562113 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.562500 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.563285 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.564111 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.564438 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.564546 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.567137 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:55815: BlockingRecv error: recv got EOF from 127.3.183.194:55815 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:21.573796 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.567814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:39755)
0929 17:48:21.567973 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.567986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.568060 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.568303 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.568305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.568433 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.568509 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.568514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.569724 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.569727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.570389 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.570402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.571061 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.571064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.571151 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.571739 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.571751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.573676 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 17:48:21.573971 12709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.567888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:39755 (local address 127.3.183.254:46745)
0929 17:48:21.567995 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.567999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.568028 (+ 29us) server_negotiation.cc:407] Connection header received
0929 17:48:21.568128 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.568132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.568172 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.568248 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.568625 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.569503 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.570575 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.570930 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.571056 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.573849 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:39755: BlockingRecv error: recv got EOF from 127.3.183.194:39755 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 17:48:21.574251 12840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:48:21.668764 12843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:21.669034 12843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:21.669076 12843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:21.673271 12843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:21.673347 12843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:21.673429 12843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:21.678787 12843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/krb5kdc/oryx_127.3.183.195.keytab
--principal=oryx/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:33519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=38213
--tserver_master_addrs=127.3.183.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:21.679885 12843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:21.680897 12843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:21.690033 12851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:21.690075 12848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:21.690075 12849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:21.690352 12843 server_base.cc:1034] running on GCE node
I20260929 17:48:21.690862 12843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:21.691515 12843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:21.692669 12843 hybrid_clock.cc:648] HybridClock initialized: now 1790704101692596 us; error 88 us; skew 500 ppm
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704101, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:21.695348 12843 init.cc:377] Logged in from keytab as oryx/127.3.183.195@KRBTEST.COM (short username oryx)
I20260929 17:48:21.696260 12843 webserver.cc:460] Webserver started at http://127.3.183.195:38213/ using document root <none> and password file <none>
I20260929 17:48:21.696787 12843 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:21.696863 12843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:21.700874 12843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:21.703118 12858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.704515 12843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:48:21.704627 12843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "4f565388728443769807efd39f1be973"
format_stamp: "Formatted at 2026-09-29 17:48:20 on dist-test-slave-wbpt"
server_key: "9561b5c4b48587ac1cd4d59c13179bd6"
server_key_iv: "bb0df8bb5a4723c44d4b0fa35eae9250"
server_key_version: "encryptionkey@0"
I20260929 17:48:21.705204 12843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:21.716288 12843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:21.719245 12843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:21.719411 12843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:21.720041 12843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:21.720844 12843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:21.720892 12843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.720932 12843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:21.720960 12843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:21.729221 12843 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:33519
I20260929 17:48:21.729247 12971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:33519 every 8 connection(s)
I20260929 17:48:21.730187 12843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:21.733276 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 12843
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:48:21.741334 12709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.731858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:60071 (local address 127.3.183.254:46745)
0929 17:48:21.732000 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.732003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.732818 (+ 815us) server_negotiation.cc:407] Connection header received
0929 17:48:21.734272 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.734275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.734330 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.734404 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.735539 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.736075 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.737191 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.737424 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.737522 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.741209 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:60071: BlockingRecv error: recv got EOF from 127.3.183.195:60071 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:48:21.742157 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.732230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:60071)
0929 17:48:21.732674 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.732714 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.734070 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.734592 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.734599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.734941 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.735402 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.735415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.736185 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.736189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.737065 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.737075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.737607 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.737611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.737809 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.738362 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.738382 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.741041 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
W20260929 17:48:21.742604 12972 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:46745 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.183.254:46745: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
I20260929 17:48:21.743180 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.734218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.3.183.254:46745)
0929 17:48:21.734436 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.734439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.734450 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:48:21.734487 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.734489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.734521 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.734592 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.735320 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.735886 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.736710 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.736928 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.737010 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.738603 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:21.738608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:21.738623 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:21.738649 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:21.740737 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:21.741273 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:21.741279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:21.741284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:21.741499 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:21.741780 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:21.741783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:21.741785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:21.742219 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:21.742408 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:21.742768 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 17:48:21.743042 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:21.747653 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.743003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:51791)
0929 17:48:21.743100 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.743114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.743183 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.743547 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.743550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.743666 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.743752 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.743756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.744538 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.744541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.745085 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.745092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.745551 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.745554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.745645 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.746008 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.746019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.747534 (+ 1515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:48:21.747758 12976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.743237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:51791 (local address 127.3.183.254:46745)
0929 17:48:21.743375 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.743379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.743390 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:48:21.743422 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.743424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.743462 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.743534 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.743851 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.744438 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.745183 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.745452 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.745540 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.747666 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:51791: BlockingRecv error: recv got EOF from 127.3.183.195:51791 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:21 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:21.752485 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:21.748165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:59561)
0929 17:48:21.748268 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:21.748279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:21.748347 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:21.748598 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:21.748601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:21.748729 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:21.748794 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.748798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.749496 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.749500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.750008 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:21.750015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.750389 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.750392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:21.750464 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.750847 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:21.750858 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:21.752389 (+ 1531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 17:48:21.752628 12976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:21.748257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:59561 (local address 127.3.183.254:46745)
0929 17:48:21.748400 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:21.748404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:21.748415 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:48:21.748445 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:21.748446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:21.748477 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:21.748572 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:21.748887 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.749402 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.750102 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:21.750301 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:21.750379 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:21.752518 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:59561: BlockingRecv error: recv got EOF from 127.3.183.195:59561 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 17:48:21.752719 12972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 17:48:22 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:22 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:22.380869 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:22.375433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:44383)
0929 17:48:22.375744 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:22.375756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:22.375875 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:22.376220 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:22.376223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:22.376393 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:22.376528 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:22.376534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.377305 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.377308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:22.377939 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:22.377949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.378343 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.378345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:22.378418 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:22.378939 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:22.378952 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:22.380751 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":73,"threads_started":1}
W20260929 17:48:22.380995 12978 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:22.375528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:44383 (local address 127.3.183.254:46745)
0929 17:48:22.375800 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:22.375804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:22.375820 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:48:22.376004 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:22.376007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:22.376060 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:22.376151 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:22.376655 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.377199 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.378033 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.378252 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.378320 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:22.380881 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:44383: BlockingRecv error: recv got EOF from 127.3.183.193:44383 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":69,"threads_started":1}
Sep 29 17:48:22 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:22 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:22.580653 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:22.574807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:49151)
0929 17:48:22.575103 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:22.575119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:22.575232 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:22.575480 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:22.575484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:22.575690 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:22.575815 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:22.575822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.576553 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.576557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:22.577127 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:22.577135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.577718 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.577722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:22.577818 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:22.578330 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:22.578344 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:22.580516 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
W20260929 17:48:22.580804 12978 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:22.574908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:49151 (local address 127.3.183.254:46745)
0929 17:48:22.575077 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:22.575082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:22.575190 (+ 108us) server_negotiation.cc:407] Connection header received
0929 17:48:22.575328 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:22.575330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:22.575373 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:22.575490 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:22.575932 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.576452 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.577247 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.577574 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.577685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:22.580687 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:49151: BlockingRecv error: recv got EOF from 127.3.183.194:49151 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 17:48:22 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:22 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:22.759289 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:22.753364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:48673)
0929 17:48:22.753666 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:22.753678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:22.753781 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:22.754052 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:22.754054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:22.754226 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:22.754346 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:22.754352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.755189 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.755193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:22.755855 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:22.755869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.756369 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.756373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:22.756453 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:22.756995 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:22.757008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:22.759139 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":122,"threads_started":1}
W20260929 17:48:22.759426 12978 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:22.753547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:48673 (local address 127.3.183.254:46745)
0929 17:48:22.753678 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:22.753682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:22.753747 (+ 65us) server_negotiation.cc:407] Connection header received
0929 17:48:22.753861 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:22.753864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:22.753937 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:22.754021 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:22.754472 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.755050 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.755996 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:22.756228 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:22.756321 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:22.759298 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:48673: BlockingRecv error: recv got EOF from 127.3.183.195:48673 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 17:48:23 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:23 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:23.386541 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:23.381581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:33647)
0929 17:48:23.381819 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:23.381832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:23.381920 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:23.382155 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:23.382156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:23.382336 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:23.382432 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:23.382436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.383202 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.383204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:23.383746 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:23.383753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.384210 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.384213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:23.384273 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:23.384723 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:23.384734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:23.386426 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
W20260929 17:48:23.386727 12982 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:23.381626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:33647 (local address 127.3.183.254:46745)
0929 17:48:23.381845 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:23.381849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:23.381878 (+ 29us) server_negotiation.cc:407] Connection header received
0929 17:48:23.381989 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:23.381991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:23.382051 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:23.382132 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:23.382555 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.383119 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.383883 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.384116 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.384195 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:23.386594 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:33647: BlockingRecv error: recv got EOF from 127.3.183.193:33647 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
Sep 29 17:48:23 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:23 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:23.586766 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:23.581596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:49391)
0929 17:48:23.581898 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:23.581910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:23.582016 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:23.582259 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:23.582262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:23.582449 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:23.582553 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:23.582559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.583288 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.583291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:23.583896 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:23.583905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.584321 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.584325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:23.584403 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:23.584924 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:23.584936 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:23.586627 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":71,"threads_started":1}
W20260929 17:48:23.586952 12982 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:23.581635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:49391 (local address 127.3.183.254:46745)
0929 17:48:23.581810 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:23.581815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:23.581980 (+ 165us) server_negotiation.cc:407] Connection header received
0929 17:48:23.582108 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:23.582111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:23.582155 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:23.582220 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:23.582665 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.583182 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.584006 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.584226 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.584310 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:23.586796 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:49391: BlockingRecv error: recv got EOF from 127.3.183.194:49391 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 17:48:23 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:23 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:23.765582 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:23.760173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:59419)
0929 17:48:23.760457 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:23.760471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:23.760571 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:23.760900 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:23.760904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:23.761134 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:23.761237 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:23.761244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.762082 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.762086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:23.762673 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:23.762683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.763102 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.763106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:23.763186 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:23.763686 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:23.763698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:23.765432 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
W20260929 17:48:23.765681 12982 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:23.760357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:59419 (local address 127.3.183.254:46745)
0929 17:48:23.760485 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:23.760488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:23.760535 (+ 47us) server_negotiation.cc:407] Connection header received
0929 17:48:23.760691 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:23.760694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:23.760744 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:23.760820 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:23.761384 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.761977 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.762791 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:23.763005 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:23.763089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:23.765581 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:59419: BlockingRecv error: recv got EOF from 127.3.183.195:59419 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 17:48:24 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:24 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:24.392899 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:24.387382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:40993)
0929 17:48:24.387642 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:24.387655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:24.387745 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:24.388009 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:24.388011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:24.388182 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:24.388280 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:24.388286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.389014 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.389017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:24.389595 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:24.389605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.390040 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.390043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:24.390115 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:24.390634 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:24.390645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:24.392734 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
W20260929 17:48:24.393118 12986 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:24.387491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:40993 (local address 127.3.183.254:46745)
0929 17:48:24.387762 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:24.387766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:24.387799 (+ 33us) server_negotiation.cc:407] Connection header received
0929 17:48:24.387845 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:24.387848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:24.387894 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:24.387974 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:24.388372 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.388901 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.389695 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.389907 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.389981 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:24.392986 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:40993: BlockingRecv error: recv got EOF from 127.3.183.193:40993 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
Sep 29 17:48:24 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:24 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:24.593312 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:24.587611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:47175)
0929 17:48:24.587918 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:24.587939 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:24.588035 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:24.588319 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:24.588322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:24.588541 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:24.588654 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:24.588660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.589687 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.589691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:24.590329 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:24.590340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.590769 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.590772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:24.590833 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:24.591309 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:24.591320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:24.593154 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
W20260929 17:48:24.593503 12986 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:24.587752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:47175 (local address 127.3.183.254:46745)
0929 17:48:24.587891 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:24.587895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:24.588013 (+ 118us) server_negotiation.cc:407] Connection header received
0929 17:48:24.588104 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:24.588106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:24.588149 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:24.588223 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:24.588799 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.589560 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.590438 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.590680 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.590757 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:24.593340 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:47175: BlockingRecv error: recv got EOF from 127.3.183.194:47175 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 17:48:24 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:24 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:24.772122 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:24.766326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:40747)
0929 17:48:24.766633 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:24.766646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:24.766741 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:24.766940 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:24.766942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:24.767117 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:24.767227 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:24.767235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.768380 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.768383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:24.768970 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:24.768980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.769454 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.769457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:24.769556 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:24.770048 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:24.770061 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:24.771954 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":83,"threads_started":1}
W20260929 17:48:24.772209 12986 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:24.766446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:40747 (local address 127.3.183.254:46745)
0929 17:48:24.766553 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:24.766557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:24.766707 (+ 150us) server_negotiation.cc:407] Connection header received
0929 17:48:24.766803 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:24.766805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:24.766848 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:24.766914 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:24.767346 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.768266 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.769093 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:24.769309 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:24.769388 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:24.772105 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:40747: BlockingRecv error: recv got EOF from 127.3.183.195:40747 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 29 17:48:25 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:25 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:25.408562 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:25.393972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:49837)
0929 17:48:25.394247 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:25.394260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:25.394346 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:25.394586 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:25.394588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:25.394765 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:25.394884 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:25.394891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.395874 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.395878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:25.396744 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:25.396755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.397347 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.397351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:25.397440 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:25.406562 (+ 9122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:25.406577 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:25.408439 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260929 17:48:25.408825 12990 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:25.394093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:49837 (local address 127.3.183.254:46745)
0929 17:48:25.394322 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:25.394325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:25.394338 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:25.394424 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:25.394426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:25.394466 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:25.394526 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:25.395010 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.395744 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.396863 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.397216 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.397314 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:25.408616 (+ 11302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:49837: BlockingRecv error: recv got EOF from 127.3.183.193:49837 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
Sep 29 17:48:25 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:25 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:25.599507 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:25.594414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:47523)
0929 17:48:25.594690 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:25.594703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:25.594816 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:25.595023 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:25.595025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:25.595195 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:25.595289 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:25.595294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.596058 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.596062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:25.596646 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:25.596655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.597124 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.597128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:25.597228 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:25.597696 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:25.597708 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:25.599382 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
W20260929 17:48:25.599623 12990 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:25.594564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:47523 (local address 127.3.183.254:46745)
0929 17:48:25.594716 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:25.594720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:25.594769 (+ 49us) server_negotiation.cc:407] Connection header received
0929 17:48:25.594889 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:25.594891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:25.594932 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:25.594996 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:25.595386 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.595958 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.596758 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.597011 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.597112 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:25.599509 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:47523: BlockingRecv error: recv got EOF from 127.3.183.194:47523 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 17:48:25 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:25 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:25.778626 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:25.773021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:47551)
0929 17:48:25.773275 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:25.773289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:25.773421 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:25.773734 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:25.773737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:25.773927 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:25.774033 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:25.774039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.774825 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.774828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:25.775392 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:25.775401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.775880 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.775884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:25.775985 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:25.776486 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:25.776500 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:25.778387 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
W20260929 17:48:25.778652 12990 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:25.773107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:47551 (local address 127.3.183.254:46745)
0929 17:48:25.773254 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:25.773258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:25.773367 (+ 109us) server_negotiation.cc:407] Connection header received
0929 17:48:25.773569 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:25.773572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:25.773623 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:25.773692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:25.774157 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.774715 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.775513 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:25.775740 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:25.775870 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:25.778551 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:47551: BlockingRecv error: recv got EOF from 127.3.183.195:47551 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 17:48:26 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:26 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:26.414503 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:26.409258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:41839)
0929 17:48:26.409564 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:26.409578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:26.409708 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:26.409961 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:26.409963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:26.410171 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:26.410270 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:26.410276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.411058 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.411061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:26.411651 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:26.411661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.412188 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.412191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:26.412251 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:26.412712 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:26.412722 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:26.414392 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
W20260929 17:48:26.414700 12994 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:26.409308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:41839 (local address 127.3.183.254:46745)
0929 17:48:26.409596 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:26.409601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:26.409636 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:48:26.409795 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:26.409798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:26.409848 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:26.409950 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:26.410361 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.410971 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.411826 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.412093 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.412179 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:26.414553 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:41839: BlockingRecv error: recv got EOF from 127.3.183.193:41839 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":127,"threads_started":1}
Sep 29 17:48:26 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:26 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:26.605907 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:26.600358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:51165)
0929 17:48:26.600686 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:26.600700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:26.600798 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:26.601023 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:26.601025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:26.601211 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:26.601312 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:26.601318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.602031 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.602034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:26.602806 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:26.602820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.603341 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.603346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:26.603443 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:26.603929 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:26.603942 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:26.605760 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":84,"threads_started":1}
W20260929 17:48:26.605983 12994 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:26.600513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:51165 (local address 127.3.183.254:46745)
0929 17:48:26.600658 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:26.600661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:26.600749 (+ 88us) server_negotiation.cc:407] Connection header received
0929 17:48:26.600876 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:26.600878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:26.600920 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:26.600989 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:26.601407 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.601913 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.602930 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.603209 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.603305 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:26.605887 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:51165: BlockingRecv error: recv got EOF from 127.3.183.194:51165 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 17:48:26 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:26 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:26.784613 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:26.779452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:43837)
0929 17:48:26.779711 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:26.779726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:26.779847 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:26.780075 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:26.780078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:26.780249 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:26.780344 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:26.780350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.781112 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.781115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:26.781741 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:26.781751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.782187 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.782190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:26.782253 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:26.782711 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:26.782723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:26.784497 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
W20260929 17:48:26.784786 12994 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:26.779535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:43837 (local address 127.3.183.254:46745)
0929 17:48:26.779691 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:26.779694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:26.779791 (+ 97us) server_negotiation.cc:407] Connection header received
0929 17:48:26.779924 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:26.779927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:26.779971 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:26.780035 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:26.780476 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.781011 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.781860 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:26.782083 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:26.782162 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:26.784662 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:43837: BlockingRecv error: recv got EOF from 127.3.183.195:43837 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 17:48:27 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:27 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:27.420117 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:27.415138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:55073)
0929 17:48:27.415379 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:27.415391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:27.415475 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:27.415689 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:27.415691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:27.415889 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:27.415987 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:27.415993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.416739 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.416742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:27.417325 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:27.417333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.417733 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.417735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:27.417795 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:27.418272 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:27.418283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:27.419979 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
W20260929 17:48:27.420207 12998 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:27.415241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:55073 (local address 127.3.183.254:46745)
0929 17:48:27.415459 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:27.415463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:27.415475 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:48:27.415549 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:27.415551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:27.415593 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:27.415667 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:27.416091 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.416636 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.417411 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.417646 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.417718 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:27.420101 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:55073: BlockingRecv error: recv got EOF from 127.3.183.193:55073 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
Sep 29 17:48:27 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:27 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:27.611958 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:27.606623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:42547)
0929 17:48:27.606902 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:27.606917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:27.607039 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:27.607275 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:27.607277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:27.607521 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:27.607616 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:27.607622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.608421 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.608425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:27.608980 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:27.608987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.609468 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.609472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:27.609558 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:27.610029 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:27.610041 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:27.611810 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
W20260929 17:48:27.612036 12998 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:27.606685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:42547 (local address 127.3.183.254:46745)
0929 17:48:27.606857 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:27.606862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:27.606990 (+ 128us) server_negotiation.cc:407] Connection header received
0929 17:48:27.607125 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:27.607128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:27.607174 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:27.607242 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:27.607728 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.608311 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.609099 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.609352 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.609438 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:27.611939 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:42547: BlockingRecv error: recv got EOF from 127.3.183.194:42547 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 17:48:27 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:27 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:27.790337 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:27.785327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:56945)
0929 17:48:27.785614 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:27.785626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:27.785727 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:27.785939 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:27.785941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:27.786132 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:27.786232 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:27.786238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.786926 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.786930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:27.787487 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:27.787497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.787910 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.787913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:27.787979 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:27.788441 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:27.788452 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:27.790192 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":68,"threads_started":1}
W20260929 17:48:27.790467 12998 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:27.785422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:56945 (local address 127.3.183.254:46745)
0929 17:48:27.785547 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:27.785550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:27.785688 (+ 138us) server_negotiation.cc:407] Connection header received
0929 17:48:27.785798 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:27.785800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:27.785844 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:27.785910 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:27.786319 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.786826 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.787604 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:27.787828 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:27.787895 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:27.790345 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:56945: BlockingRecv error: recv got EOF from 127.3.183.195:56945 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 17:48:28 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:28 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:28.425881 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:28.420859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:34837)
0929 17:48:28.421173 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:28.421187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:28.421286 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:28.421497 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:28.421499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:28.421673 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:28.421776 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:28.421781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.422493 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.422496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:28.423066 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:28.423076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.423504 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.423507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:28.423607 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:28.424119 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:28.424131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:28.425764 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":81,"threads_started":1}
W20260929 17:48:28.426007 13002 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:28.420973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:34837 (local address 127.3.183.254:46745)
0929 17:48:28.421206 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:28.421209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:28.421241 (+ 32us) server_negotiation.cc:407] Connection header received
0929 17:48:28.421351 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:28.421353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:28.421393 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:28.421466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:28.421864 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.422386 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.423176 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.423400 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.423473 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:28.425892 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:34837: BlockingRecv error: recv got EOF from 127.3.183.193:34837 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":60,"threads_started":1}
Sep 29 17:48:28 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:28 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:28.618461 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:28.612781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:39963)
0929 17:48:28.613156 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:28.613170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:28.613288 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:28.613551 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:28.613554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:28.613750 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:28.613841 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:28.613847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.614721 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.614725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:28.615469 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:28.615479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.615944 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.615947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:28.616024 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:28.616499 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:28.616511 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:28.618339 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":73,"threads_started":1}
W20260929 17:48:28.618579 13002 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:28.612877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:39963 (local address 127.3.183.254:46745)
0929 17:48:28.613100 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:28.613105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:28.613247 (+ 142us) server_negotiation.cc:407] Connection header received
0929 17:48:28.613381 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:28.613384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:28.613427 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:28.613505 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:28.613962 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.614622 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.615572 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.615848 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.615945 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:28.618480 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:39963: BlockingRecv error: recv got EOF from 127.3.183.194:39963 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 17:48:28 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:28 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:28.797083 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:28.791147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:33269)
0929 17:48:28.791478 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:28.791495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:28.791618 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:28.791936 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:28.791939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:28.792153 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:28.792262 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:28.792269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.793113 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.793118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:28.793806 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:28.793817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.794282 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.794286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:28.794374 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:28.794898 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:28.794911 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:28.796957 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":78,"threads_started":1}
W20260929 17:48:28.797237 13002 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:28.791272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:33269 (local address 127.3.183.254:46745)
0929 17:48:28.791433 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:28.791438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:28.791569 (+ 131us) server_negotiation.cc:407] Connection header received
0929 17:48:28.791726 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:28.791729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:28.791800 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:28.791870 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:28.792392 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.792984 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.793938 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:28.794170 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:28.794267 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:28.797110 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:33269: BlockingRecv error: recv got EOF from 127.3.183.195:33269 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 17:48:29 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:29 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:29.432193 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:29.426835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:51363)
0929 17:48:29.427136 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:29.427151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:29.427254 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:29.427523 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:29.427526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:29.427727 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:29.427866 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:29.427872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.428634 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.428638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:29.429200 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:29.429208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.429635 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.429639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:29.429720 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:29.430210 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:29.430222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:29.432040 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
W20260929 17:48:29.432315 13006 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:29.426983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:51363 (local address 127.3.183.254:46745)
0929 17:48:29.427235 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:29.427240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:29.427253 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:29.427361 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:29.427364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:29.427408 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:29.427480 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:29.427987 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.428527 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.429312 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.429536 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.429620 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:29.432201 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:51363: BlockingRecv error: recv got EOF from 127.3.183.193:51363 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
Sep 29 17:48:29 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:29 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:29.624560 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:29.619222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:59387)
0929 17:48:29.619461 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:29.619475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:29.619583 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:29.619919 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:29.619921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:29.620113 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:29.620207 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:29.620213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.621083 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.621087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:29.621647 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:29.621654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.622086 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.622089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:29.622149 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:29.622597 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:29.622608 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:29.624409 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
W20260929 17:48:29.624675 13006 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:29.619334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:59387 (local address 127.3.183.254:46745)
0929 17:48:29.619489 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:29.619493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:29.619545 (+ 52us) server_negotiation.cc:407] Connection header received
0929 17:48:29.619743 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:29.619746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:29.619821 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:29.619894 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:29.620309 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.620981 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.621755 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.621982 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.622061 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:29.624542 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:59387: BlockingRecv error: recv got EOF from 127.3.183.194:59387 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 17:48:29 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:29 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:29.802949 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:29.797898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:38039)
0929 17:48:29.798155 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:29.798168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:29.798264 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:29.798476 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:29.798478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:29.798659 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:29.798755 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:29.798760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.799480 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.799483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:29.800134 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:29.800144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.800557 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.800559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:29.800613 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:29.801051 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:29.801061 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:29.802803 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
W20260929 17:48:29.803064 13006 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:29.797998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:38039 (local address 127.3.183.254:46745)
0929 17:48:29.798135 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:29.798139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:29.798229 (+ 90us) server_negotiation.cc:407] Connection header received
0929 17:48:29.798342 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:29.798344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:29.798388 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:29.798448 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:29.798865 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.799389 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.800253 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:29.800469 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:29.800549 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:29.802942 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:38039: BlockingRecv error: recv got EOF from 127.3.183.195:38039 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 17:48:30 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:30 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:30.441398 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:30.432986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:34971)
0929 17:48:30.433260 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:30.433274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:30.433366 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:30.433588 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:30.433590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:30.433760 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:30.433856 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:30.433860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.434536 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.434538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:30.435093 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:30.435100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.435510 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.435515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:30.435583 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:30.439257 (+ 3674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:30.439274 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:30.441264 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
W20260929 17:48:30.441515 13010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:30.433091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:34971 (local address 127.3.183.254:46745)
0929 17:48:30.433321 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:30.433325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:30.433337 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:48:30.433436 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:30.433438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:30.433479 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:30.433549 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:30.433938 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.434451 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.435188 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.435408 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.435482 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:30.441403 (+ 5921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:34971: BlockingRecv error: recv got EOF from 127.3.183.193:34971 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
Sep 29 17:48:30 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:30 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:30.630383 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:30.625373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:44937)
0929 17:48:30.625664 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:30.625676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:30.625770 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:30.626027 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:30.626030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:30.626211 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:30.626311 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:30.626316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.626998 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.627001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:30.627573 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:30.627581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.628001 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.628003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:30.628097 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:30.628519 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:30.628529 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:30.630253 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":101,"threads_started":1}
W20260929 17:48:30.630573 13010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:30.625451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:44937 (local address 127.3.183.254:46745)
0929 17:48:30.625605 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:30.625609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:30.625750 (+ 141us) server_negotiation.cc:407] Connection header received
0929 17:48:30.625841 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:30.625844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:30.625890 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:30.625958 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:30.626400 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.626917 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.627681 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.627918 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.627995 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:30.630400 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:44937: BlockingRecv error: recv got EOF from 127.3.183.194:44937 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 17:48:30 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:30 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:30.809477 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:30.803867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:33737)
0929 17:48:30.804180 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:30.804195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:30.804306 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:30.804550 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:30.804553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:30.804759 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:30.804858 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:30.804864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.805686 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.805690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:30.806241 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:30.806251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.806667 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.806671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:30.806753 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:30.807224 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:30.807236 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:30.809302 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":86,"threads_started":1}
W20260929 17:48:30.809564 13010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:30.803957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:33737 (local address 127.3.183.254:46745)
0929 17:48:30.804139 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:30.804143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:30.804263 (+ 120us) server_negotiation.cc:407] Connection header received
0929 17:48:30.804394 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:30.804397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:30.804445 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:30.804513 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:30.804984 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.805568 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.806352 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:30.806566 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:30.806652 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:30.809443 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:33737: BlockingRecv error: recv got EOF from 127.3.183.195:33737 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 17:48:31 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:31 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:31.447382 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:31.442165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:47285)
0929 17:48:31.442433 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:31.442447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:31.442537 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:31.442804 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:31.442807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:31.442979 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:31.443082 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:31.443088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.443965 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.443968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:31.444557 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:31.444567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.444989 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.444992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:31.445052 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:31.445483 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:31.445493 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:31.447253 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
W20260929 17:48:31.447539 13014 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:31.442241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:47285 (local address 127.3.183.254:46745)
0929 17:48:31.442482 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:31.442486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:31.442499 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:31.442633 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:31.442636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:31.442681 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:31.442756 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:31.443222 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.443838 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.444680 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.444903 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.444983 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:31.447408 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:47285: BlockingRecv error: recv got EOF from 127.3.183.193:47285 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
Sep 29 17:48:31 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:31 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:31.636386 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:31.631072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:42949)
0929 17:48:31.631340 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:31.631356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:31.631479 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:31.631757 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:31.631760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:31.631981 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:31.632104 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:31.632110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.632845 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.632849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:31.633417 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:31.633425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.633848 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.633851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:31.633933 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:31.634435 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:31.634447 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:31.636272 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
W20260929 17:48:31.636488 13014 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:31.631203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:42949 (local address 127.3.183.254:46745)
0929 17:48:31.631342 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:31.631346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:31.631436 (+ 90us) server_negotiation.cc:407] Connection header received
0929 17:48:31.631600 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:31.631603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:31.631646 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:31.631711 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:31.632220 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.632740 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.633534 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.633749 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.633834 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:31.636395 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:42949: BlockingRecv error: recv got EOF from 127.3.183.194:42949 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 17:48:31 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:31 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:31.815541 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:31.810240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:57263)
0929 17:48:31.810487 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:31.810502 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:31.810650 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:31.810881 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:31.810883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:31.811050 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:31.811159 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:31.811165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.811979 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.811981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:31.812574 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:31.812584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.813012 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.813016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:31.813109 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:31.813612 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:31.813624 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:31.815416 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
W20260929 17:48:31.815698 13014 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:31.810373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:57263 (local address 127.3.183.254:46745)
0929 17:48:31.810487 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:31.810491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:31.810586 (+ 95us) server_negotiation.cc:407] Connection header received
0929 17:48:31.810731 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:31.810733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:31.810787 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:31.810850 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:31.811272 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.811881 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.812673 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:31.812893 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:31.812970 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:31.815586 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:57263: BlockingRecv error: recv got EOF from 127.3.183.195:57263 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 17:48:32 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:32 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:32.453507 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:32.448193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:57637)
0929 17:48:32.448472 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:32.448485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:32.448573 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:32.448793 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:32.448796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:32.448970 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:32.449071 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:32.449076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.449835 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.449837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:32.450402 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:32.450410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.450860 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.450862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:32.450918 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:32.451417 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:32.451429 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:32.453389 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
W20260929 17:48:32.453685 13018 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:32.448327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:57637 (local address 127.3.183.254:46745)
0929 17:48:32.448533 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:32.448536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:32.448549 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:32.448663 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:32.448665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:32.448708 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:32.448782 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:32.449177 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.449728 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.450532 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.450767 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.450853 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:32.453552 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:57637: BlockingRecv error: recv got EOF from 127.3.183.193:57637 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
Sep 29 17:48:32 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:32 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:32.642759 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:32.637215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:57815)
0929 17:48:32.637470 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:32.637483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:32.637580 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:32.637800 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:32.637802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:32.637976 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:32.638096 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:32.638102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.638884 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.638887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:32.639563 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:32.639577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.640116 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.640119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:32.640185 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:32.640724 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:32.640736 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:32.642632 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
W20260929 17:48:32.642881 13018 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:32.637286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:57815 (local address 127.3.183.254:46745)
0929 17:48:32.637413 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:32.637416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:32.637571 (+ 155us) server_negotiation.cc:407] Connection header received
0929 17:48:32.637651 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:32.637654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:32.637699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:32.637768 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:32.638192 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.638767 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.639712 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.639985 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.640080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:32.642780 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:57815: BlockingRecv error: recv got EOF from 127.3.183.194:57815 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 17:48:32 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:32 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:32.821365 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:32.816270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:51941)
0929 17:48:32.816519 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:32.816532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:32.816637 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:32.816848 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:32.816850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:32.817017 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:32.817138 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:32.817155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.817895 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.817898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:32.818486 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:32.818494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.818909 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.818912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:32.818969 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:32.819424 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:32.819436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:32.821216 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
W20260929 17:48:32.821456 13018 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:32.816367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:51941 (local address 127.3.183.254:46745)
0929 17:48:32.816483 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:32.816487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:32.816594 (+ 107us) server_negotiation.cc:407] Connection header received
0929 17:48:32.816704 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:32.816707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:32.816751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:32.816824 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:32.817238 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.817799 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.818606 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:32.818822 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:32.818896 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:32.821354 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:51941: BlockingRecv error: recv got EOF from 127.3.183.195:51941 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 17:48:33 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:33 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:33.459599 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:33.454277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:44097)
0929 17:48:33.454565 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:33.454578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:33.454668 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:33.454890 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:33.454892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:33.455083 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:33.455178 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:33.455183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.455984 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.455987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:33.456749 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:33.456761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.457280 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.457283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:33.457344 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:33.457790 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:33.457801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:33.459483 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
W20260929 17:48:33.459745 13022 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:33.454352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:44097 (local address 127.3.183.254:46745)
0929 17:48:33.454608 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:33.454611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:33.454626 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:48:33.454745 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:33.454747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:33.454789 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:33.454869 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:33.455267 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.455893 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.456857 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.457178 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.457260 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:33.459616 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:44097: BlockingRecv error: recv got EOF from 127.3.183.193:44097 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
Sep 29 17:48:33 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:33 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:33.648703 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:33.643439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:55551)
0929 17:48:33.643726 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:33.643740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:33.643882 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:33.644136 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:33.644139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:33.644331 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:33.644425 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:33.644431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.645179 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.645183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:33.645760 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:33.645769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.646204 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.646208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:33.646292 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:33.646751 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:33.646762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:33.648552 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
W20260929 17:48:33.648804 13022 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:33.643509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:55551 (local address 127.3.183.254:46745)
0929 17:48:33.643685 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:33.643690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:33.643814 (+ 124us) server_negotiation.cc:407] Connection header received
0929 17:48:33.643974 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:33.643977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:33.644024 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:33.644095 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:33.644546 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.645076 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.645877 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.646104 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.646187 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:33.648705 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:55551: BlockingRecv error: recv got EOF from 127.3.183.194:55551 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 17:48:33 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:33 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:33.827188 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:33.822069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:48805)
0929 17:48:33.822367 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:33.822379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:33.822472 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:33.822698 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:33.822700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:33.822872 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:33.822970 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:33.822975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.823672 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.823675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:33.824267 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:33.824276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.824680 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.824682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:33.824737 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:33.825193 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:33.825205 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:33.827068 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
W20260929 17:48:33.827266 13022 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:33.822175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:48805 (local address 127.3.183.254:46745)
0929 17:48:33.822321 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:33.822324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:33.822441 (+ 117us) server_negotiation.cc:407] Connection header received
0929 17:48:33.822548 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:33.822550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:33.822597 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:33.822672 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:33.823065 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.823585 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.824361 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:33.824593 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:33.824669 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:33.827173 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:48805: BlockingRecv error: recv got EOF from 127.3.183.195:48805 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 17:48:34 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:34 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:34.466058 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:34.460380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:33143)
0929 17:48:34.460719 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:34.460732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:34.460830 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:34.461176 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:34.461178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:34.461369 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:34.461526 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:34.461533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.462290 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.462293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:34.462879 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:34.462888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.463356 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.463359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:34.463434 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:34.463953 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:34.463968 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:34.465901 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":77,"threads_started":1}
W20260929 17:48:34.466195 13026 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:34.460474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:33143 (local address 127.3.183.254:46745)
0929 17:48:34.460728 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:34.460732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:34.460789 (+ 57us) server_negotiation.cc:407] Connection header received
0929 17:48:34.460982 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:34.460985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:34.461032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:34.461113 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:34.461654 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.462192 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.463003 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.463230 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.463314 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:34.466077 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:33143: BlockingRecv error: recv got EOF from 127.3.183.193:33143 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
Sep 29 17:48:34 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:34 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:34.654744 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:34.649499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:42433)
0929 17:48:34.649743 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:34.649758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:34.649849 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:34.650169 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:34.650172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:34.650355 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:34.650455 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:34.650462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.651193 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.651196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:34.651807 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:34.651816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.652231 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.652234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:34.652299 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:34.652799 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:34.652811 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:34.654611 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":64,"threads_started":1}
W20260929 17:48:34.654856 13026 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:34.649661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:42433 (local address 127.3.183.254:46745)
0929 17:48:34.649796 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:34.649800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:34.649813 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:34.649947 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:34.649949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:34.650007 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:34.650109 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:34.650557 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.651087 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.651913 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.652140 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.652216 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:34.654755 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:42433: BlockingRecv error: recv got EOF from 127.3.183.194:42433 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 17:48:34 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:34 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:34.833662 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:34.827976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:38739)
0929 17:48:34.828278 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:34.828291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:34.828408 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:34.828669 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:34.828672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:34.828868 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:34.828963 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:34.828970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.829773 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.829777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:34.830329 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:34.830338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.830792 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.830796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:34.830881 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:34.831327 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:34.831339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:34.833454 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
W20260929 17:48:34.833712 13026 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:34.828086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:38739 (local address 127.3.183.254:46745)
0929 17:48:34.828232 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:34.828236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:34.828366 (+ 130us) server_negotiation.cc:407] Connection header received
0929 17:48:34.828496 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:34.828499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:34.828554 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:34.828621 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:34.829097 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.829654 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.830442 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:34.830663 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:34.830749 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:34.833609 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:38739: BlockingRecv error: recv got EOF from 127.3.183.195:38739 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 17:48:35 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:35 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:35.481364 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:35.466839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:44833)
0929 17:48:35.467102 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:35.467115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:35.467201 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:35.467476 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:35.467479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:35.467671 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:35.467767 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:35.467772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.468600 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.468603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:35.469190 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:35.469200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.469626 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.469629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:35.469700 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:35.479115 (+ 9415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:35.479131 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:35.481234 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
W20260929 17:48:35.481525 13030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:35.466956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:44833 (local address 127.3.183.254:46745)
0929 17:48:35.467273 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:35.467277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:35.467290 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:48:35.467328 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:35.467330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:35.467371 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:35.467449 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:35.467888 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.468446 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.469290 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.469504 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.469606 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:35.481394 (+ 11788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:44833: BlockingRecv error: recv got EOF from 127.3.183.193:44833 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":87,"threads_started":1}
Sep 29 17:48:35 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:35 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:35.661036 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:35.655573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:55945)
0929 17:48:35.655853 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:35.655866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:35.655956 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:35.656213 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:35.656216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:35.656395 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:35.656491 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:35.656496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.657392 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.657395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:35.658022 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:35.658031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.658565 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.658568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:35.658640 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:35.659126 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:35.659138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:35.660904 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
W20260929 17:48:35.661186 13030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:35.655715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:55945 (local address 127.3.183.254:46745)
0929 17:48:35.655889 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:35.655893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:35.655911 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:48:35.656037 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:35.656040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:35.656081 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:35.656151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:35.656667 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.657232 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.658158 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.658416 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.658512 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:35.661064 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:55945: BlockingRecv error: recv got EOF from 127.3.183.194:55945 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 17:48:35 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:35 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.195@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:35.840282 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:35.834549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.195:36149)
0929 17:48:35.834849 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:35.834865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:35.834982 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:35.835234 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:35.835237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:35.835444 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:35.835575 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:35.835582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.836475 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.836479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:35.837047 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:35.837056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.837539 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.837543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:35.837630 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:35.838101 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:35.838113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:35.840119 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":105,"threads_started":1}
W20260929 17:48:35.840412 13030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:35.834691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:36149 (local address 127.3.183.254:46745)
0929 17:48:35.834850 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:35.834854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:35.834937 (+ 83us) server_negotiation.cc:407] Connection header received
0929 17:48:35.835086 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:35.835088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:35.835133 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:35.835198 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:35.835701 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.836366 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.837177 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:35.837403 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:35.837491 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:35.840292 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.195:36149: BlockingRecv error: recv got EOF from 127.3.183.195:36149 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 17:48:36 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:36 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.193@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:36.487462 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:36.482188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.193:59339)
0929 17:48:36.482495 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:36.482510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:36.482612 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:36.482842 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:36.482844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:36.483030 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:36.483128 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:36.483133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.483927 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.483930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:36.484493 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:36.484501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.484962 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.484965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:36.485033 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:36.485500 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:36.485512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:36.487300 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
W20260929 17:48:36.487713 13034 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:36.482271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:59339 (local address 127.3.183.254:46745)
0929 17:48:36.482521 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:36.482525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:36.482563 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:48:36.482687 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:36.482689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:36.482732 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:36.482801 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:36.483218 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.483801 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.484604 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.484829 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.484910 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:36.487448 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.193:59339: BlockingRecv error: recv got EOF from 127.3.183.193:59339 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
Sep 29 17:48:36 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:48:36 dist-test-slave-wbpt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.183.194@KRBTEST.COM for oryx/127.3.183.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:48:36.667249 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:36.661974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46745 (local address 127.3.183.194:58305)
0929 17:48:36.662247 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:36.662262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:36.662370 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:36.662621 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:36.662624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:36.662817 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:36.662909 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:36.662915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.663681 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.663685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:36.664310 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:36.664320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.664748 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.664751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:36.664836 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:36.665291 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:36.665302 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:36.667114 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.183.254:46745: Server oryx/127.3.183.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260929 17:48:36.667536 13034 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:36.662101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:58305 (local address 127.3.183.254:46745)
0929 17:48:36.662249 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:36.662253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:36.662332 (+ 79us) server_negotiation.cc:407] Connection header received
0929 17:48:36.662461 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:36.662464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:36.662512 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:36.662580 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:36.663026 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.663565 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.664424 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.664646 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.664733 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:36.667277 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.183.194:58305: BlockingRecv error: recv got EOF from 127.3.183.194:58305 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:48:36.738116 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 12559
I20260929 17:48:36.743042 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 12711
I20260929 17:48:36.747540 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 12843
I20260929 17:48:36.752099 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 12489
2026-09-29T17:48:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17905 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T17:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:36Z Disabled control of system clock
I20260929 17:48:36.781015 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:33227
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 17:48:36.890286 13039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:36.890559 13039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:36.890596 13039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:36.894579 13039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:36.894706 13039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:36.894752 13039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:36.894778 13039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:36.894795 13039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:36.899562 13039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:33227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:36.900615 13039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:36.901938 13039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:36.910382 13045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:36.912216 13044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:36.912274 13039 server_base.cc:1034] running on GCE node
W20260929 17:48:36.912513 13047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:36.913053 13039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:36.914193 13039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:36.915357 13039 hybrid_clock.cc:648] HybridClock initialized: now 1790704116915341 us; error 29 us; skew 500 ppm
I20260929 17:48:36.917151 13039 webserver.cc:460] Webserver started at http://127.3.183.254:38073/ using document root <none> and password file <none>
I20260929 17:48:36.917696 13039 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:36.917761 13039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:36.917948 13039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:36.919971 13039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "7af4832120c64661b5d679cc6ebb23ca"
format_stamp: "Formatted at 2026-09-29 17:48:36 on dist-test-slave-wbpt"
server_key: "eec710ea537e1c2cce9cb8ce24759dac"
server_key_iv: "e47dd0451274d7c53211a2139bbed183"
server_key_version: "encryptionkey@0"
I20260929 17:48:36.920506 13039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "7af4832120c64661b5d679cc6ebb23ca"
format_stamp: "Formatted at 2026-09-29 17:48:36 on dist-test-slave-wbpt"
server_key: "eec710ea537e1c2cce9cb8ce24759dac"
server_key_iv: "e47dd0451274d7c53211a2139bbed183"
server_key_version: "encryptionkey@0"
I20260929 17:48:36.924306 13039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:48:36.926810 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:36.928079 13039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:36.928186 13039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "7af4832120c64661b5d679cc6ebb23ca"
format_stamp: "Formatted at 2026-09-29 17:48:36 on dist-test-slave-wbpt"
server_key: "eec710ea537e1c2cce9cb8ce24759dac"
server_key_iv: "e47dd0451274d7c53211a2139bbed183"
server_key_version: "encryptionkey@0"
I20260929 17:48:36.928277 13039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:36.943548 13039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:36.944272 13039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:36.944447 13039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:36.951242 13039 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:33227
I20260929 17:48:36.951253 13105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:33227 every 8 connection(s)
I20260929 17:48:36.952193 13039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:36.954996 13106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:36.956339 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13039
I20260929 17:48:36.956463 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:36.956772 3807 external_mini_cluster.cc:1428] Setting key c4ed3ac079543606e4b692e40e5fb786
I20260929 17:48:36.961316 13106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca: Bootstrap starting.
I20260929 17:48:36.963841 13106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:36.964556 13106 log.cc:824] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:36.966089 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:36.958274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.3.183.254:33227)
0929 17:48:36.958930 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:36.958947 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:36.958985 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:48:36.959648 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:36.959670 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:36.960098 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:36.960375 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:36.960726 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.961751 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.962900 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:36.963363 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:36.963481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:36.964353 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:36.964358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:36.964371 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:36.964800 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:36.964861 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:36.964890 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:36.964985 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:36.965131 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 17:48:36.965389 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260929 17:48:36.966497 13106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca: No bootstrap required, opened a new log
I20260929 17:48:36.970228 13106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4832120c64661b5d679cc6ebb23ca" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33227 } }
I20260929 17:48:36.970448 13106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:36.970485 13106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af4832120c64661b5d679cc6ebb23ca, State: Initialized, Role: FOLLOWER
I20260929 17:48:36.970911 13106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4832120c64661b5d679cc6ebb23ca" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33227 } }
I20260929 17:48:36.971045 13106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:36.971108 13106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:36.971201 13106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:36.972158 13106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4832120c64661b5d679cc6ebb23ca" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33227 } }
I20260929 17:48:36.972488 13106 leader_election.cc:302] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af4832120c64661b5d679cc6ebb23ca; no voters:
I20260929 17:48:36.972780 13106 leader_election.cc:288] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:36.972878 13111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:36.973100 13111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [term 1 LEADER]: Becoming Leader. State: Replica: 7af4832120c64661b5d679cc6ebb23ca, State: Running, Role: LEADER
I20260929 17:48:36.973445 13111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4832120c64661b5d679cc6ebb23ca" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33227 } }
I20260929 17:48:36.974139 13106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:36.975165 13113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af4832120c64661b5d679cc6ebb23ca. Latest consensus state: current_term: 1 leader_uuid: "7af4832120c64661b5d679cc6ebb23ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4832120c64661b5d679cc6ebb23ca" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33227 } } }
I20260929 17:48:36.975291 13113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:36.975636 13112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af4832120c64661b5d679cc6ebb23ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4832120c64661b5d679cc6ebb23ca" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33227 } } }
I20260929 17:48:36.975687 13120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:36.976266 13112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:36.978853 13120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:36.984392 13120 catalog_manager.cc:1347] Generated new cluster ID: cc395058bf5342d0976bd0f689b76d43
I20260929 17:48:36.984462 13120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:36.992761 13120 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:36.994073 13120 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:37.000561 13120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af4832120c64661b5d679cc6ebb23ca: Generated new TSK 0
I20260929 17:48:37.001431 13120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:48:37.008442 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33227
--builtin_ntp_servers=127.3.183.212:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 17:48:37.118552 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:37.118811 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:37.118863 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:37.122892 13130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:37.122951 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:37.123024 13130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:37.127702 13130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:37.128782 13130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:37.130133 13130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:37.139458 13135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:37.139619 13138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.139590 13130 server_base.cc:1034] running on GCE node
W20260929 17:48:37.139690 13136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.140169 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:37.140779 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:37.141947 13130 hybrid_clock.cc:648] HybridClock initialized: now 1790704117141924 us; error 37 us; skew 500 ppm
I20260929 17:48:37.143772 13130 webserver.cc:460] Webserver started at http://127.3.183.193:36933/ using document root <none> and password file <none>
I20260929 17:48:37.144358 13130 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:37.144419 13130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:37.144578 13130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:37.146516 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "c2e3348f0fc04338b96ae074c8437eb3"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "4cdd47582a3b7937d37521b865707e5d"
server_key_iv: "6ea91511b865dab174cb82f2f11580c6"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.147006 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "c2e3348f0fc04338b96ae074c8437eb3"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "4cdd47582a3b7937d37521b865707e5d"
server_key_iv: "6ea91511b865dab174cb82f2f11580c6"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.150511 13130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:48:37.153017 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.154388 13130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:37.154502 13130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "c2e3348f0fc04338b96ae074c8437eb3"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "4cdd47582a3b7937d37521b865707e5d"
server_key_iv: "6ea91511b865dab174cb82f2f11580c6"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.154613 13130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:37.174698 13130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:37.175479 13130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:37.175665 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:37.176294 13130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:37.177268 13130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:37.177313 13130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.177366 13130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:37.177395 13130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.185904 13130 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:41265
I20260929 17:48:37.185981 13257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:41265 every 8 connection(s)
I20260929 17:48:37.186851 13130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:37.195144 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13130
I20260929 17:48:37.195258 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:37.195573 3807 external_mini_cluster.cc:1428] Setting key 66f76d720011531df95f0b924f5a5477
I20260929 17:48:37.196411 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.188723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:40677 (local address 127.3.183.254:33227)
0929 17:48:37.188896 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:37.188899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:37.190024 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 17:48:37.191425 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:37.191428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:37.191492 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:37.191569 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:37.192560 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.193098 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.194243 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.194462 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.194552 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:37.195890 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:37.195893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:37.195896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:37.195909 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:37.195932 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:37.195941 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:37.196016 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:37.196206 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 17:48:37.196271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 17:48:37.197477 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.189082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33227 (local address 127.3.183.193:40677)
0929 17:48:37.189845 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:37.189890 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:37.191200 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:37.191767 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:37.191800 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:37.191901 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:37.192412 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:37.192425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.193232 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.193235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:37.194115 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:37.194124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.194615 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.194619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:37.194783 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:37.195542 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:37.195544 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:37.195600 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:37.195719 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:37.196058 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:37.196068 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:37.196074 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:48:37.196198 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 17:48:37.196586 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":118,"threads_started":1}
I20260929 17:48:37.197515 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33227
--builtin_ntp_servers=127.3.183.212:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 17:48:37.198674 13258 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33227
I20260929 17:48:37.198894 13258 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:37.199412 13258 heartbeater.cc:511] Master 127.3.183.254:33227 requested a full tablet report, sending...
I20260929 17:48:37.200826 13070 ts_manager.cc:194] Registered new tserver with Master: c2e3348f0fc04338b96ae074c8437eb3 (127.3.183.193:41265)
I20260929 17:48:37.201956 13070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.193:40677
W20260929 17:48:37.307083 13261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:37.307359 13261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:37.307415 13261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:37.311542 13261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:37.311602 13261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:37.311683 13261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:37.316543 13261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:37.317658 13261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:37.319025 13261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:37.328312 13266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:37.328372 13267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.328418 13261 server_base.cc:1034] running on GCE node
W20260929 17:48:37.328328 13269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.328959 13261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:37.329547 13261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:37.330735 13261 hybrid_clock.cc:648] HybridClock initialized: now 1790704117330717 us; error 36 us; skew 500 ppm
I20260929 17:48:37.332548 13261 webserver.cc:460] Webserver started at http://127.3.183.194:40043/ using document root <none> and password file <none>
I20260929 17:48:37.333092 13261 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:37.333158 13261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:37.333343 13261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:37.335301 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "11ed33a0d4a8423bb54ec578ed81b433"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "7657f8d79db1693bea8ec0cb7d437e9d"
server_key_iv: "319501e8d80125228cfe5430a32d6f43"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.335862 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "11ed33a0d4a8423bb54ec578ed81b433"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "7657f8d79db1693bea8ec0cb7d437e9d"
server_key_iv: "319501e8d80125228cfe5430a32d6f43"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.339237 13261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:48:37.341724 13275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.342864 13261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:37.342962 13261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "11ed33a0d4a8423bb54ec578ed81b433"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "7657f8d79db1693bea8ec0cb7d437e9d"
server_key_iv: "319501e8d80125228cfe5430a32d6f43"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.343055 13261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:37.355379 13261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:37.356001 13261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:37.356156 13261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:37.356690 13261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:37.357586 13261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:37.357628 13261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.357677 13261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:37.357707 13261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.365624 13261 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:45849
I20260929 17:48:37.365648 13388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:45849 every 8 connection(s)
I20260929 17:48:37.366429 13261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:37.373728 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13261
I20260929 17:48:37.373840 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:37.374188 3807 external_mini_cluster.cc:1428] Setting key 5c7dd2fdb79b4311c0a4eae1576954b7
I20260929 17:48:37.375237 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.368102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:47087 (local address 127.3.183.254:33227)
0929 17:48:37.368271 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:37.368275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:37.369460 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 17:48:37.370751 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:37.370753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:37.370800 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:37.370870 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:37.371655 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.372177 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.373264 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.373474 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.373577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:37.374629 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:37.374632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:37.374635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:37.374646 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:37.374735 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:37.374762 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:37.374846 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:37.375042 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 17:48:37.375098 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:48:37.376140 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.368544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33227 (local address 127.3.183.194:47087)
0929 17:48:37.369322 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:37.369363 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:37.370557 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:37.370996 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:37.371003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:37.371089 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:37.371544 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:37.371556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.372273 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.372276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:37.373170 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:37.373179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.373602 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.373605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:37.373733 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:37.374323 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:37.374325 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:37.374362 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:37.374421 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:37.374846 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:37.374854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:37.374860 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:48:37.374973 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 17:48:37.375213 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
I20260929 17:48:37.376341 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33227
--builtin_ntp_servers=127.3.183.212:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 17:48:37.377214 13389 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33227
I20260929 17:48:37.377457 13389 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:37.378011 13389 heartbeater.cc:511] Master 127.3.183.254:33227 requested a full tablet report, sending...
I20260929 17:48:37.379029 13070 ts_manager.cc:194] Registered new tserver with Master: 11ed33a0d4a8423bb54ec578ed81b433 (127.3.183.194:45849)
I20260929 17:48:37.379737 13070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.194:47087
W20260929 17:48:37.484715 13392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:37.484956 13392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:37.484989 13392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:37.488943 13392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:37.488996 13392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:37.489044 13392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:37.493584 13392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:37.494527 13392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:37.495847 13392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:37.504961 13398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:37.504993 13400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:37.504993 13397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.505133 13392 server_base.cc:1034] running on GCE node
I20260929 17:48:37.505527 13392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:37.506088 13392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:37.507247 13392 hybrid_clock.cc:648] HybridClock initialized: now 1790704117507231 us; error 34 us; skew 500 ppm
I20260929 17:48:37.508953 13392 webserver.cc:460] Webserver started at http://127.3.183.195:38207/ using document root <none> and password file <none>
I20260929 17:48:37.509474 13392 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:37.509539 13392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:37.509725 13392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:37.511677 13392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "f315be7dae4a461db76a509f1e507689"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "3361b71256a8603f5a9ff2fe898643a6"
server_key_iv: "0161cdfc9db033fc9cf4588bc03fe70c"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.512229 13392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "f315be7dae4a461db76a509f1e507689"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "3361b71256a8603f5a9ff2fe898643a6"
server_key_iv: "0161cdfc9db033fc9cf4588bc03fe70c"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.515898 13392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:37.518352 13406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.519620 13392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:37.519730 13392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "f315be7dae4a461db76a509f1e507689"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "3361b71256a8603f5a9ff2fe898643a6"
server_key_iv: "0161cdfc9db033fc9cf4588bc03fe70c"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.519858 13392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:37.537943 13392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:37.538625 13392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:37.538789 13392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:37.539354 13392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:37.540337 13392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:37.540385 13392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.540437 13392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:37.540467 13392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.548602 13392 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:38577
I20260929 17:48:37.548637 13519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:38577 every 8 connection(s)
I20260929 17:48:37.549453 13392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:37.552389 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13392
I20260929 17:48:37.552495 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:37.552780 3807 external_mini_cluster.cc:1428] Setting key 194b9d387c824a1570b5d8d4a3ac698c
I20260929 17:48:37.558676 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.551297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:50247 (local address 127.3.183.254:33227)
0929 17:48:37.551467 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:37.551472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:37.552555 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 17:48:37.554213 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:37.554216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:37.554271 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:37.554351 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:37.555273 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.555827 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.556931 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.557173 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.557274 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:37.558270 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:37.558273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:37.558275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:37.558288 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:37.558311 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:37.558320 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:37.558385 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:37.558506 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 17:48:37.558561 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 17:48:37.559522 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.551688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33227 (local address 127.3.183.195:50247)
0929 17:48:37.552388 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:37.552442 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:37.553999 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:37.554484 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:37.554490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:37.554581 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:37.555149 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:37.555163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.555924 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.555927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:37.556788 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:37.556796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:37.557304 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:37.557306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:37.557440 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:37.558019 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:37.558021 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:37.558058 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:37.558123 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:37.558392 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:37.558397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:37.558401 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:48:37.558490 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:48:37.558718 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
I20260929 17:48:37.560546 13520 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33227
I20260929 17:48:37.560756 13520 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:37.561239 13520 heartbeater.cc:511] Master 127.3.183.254:33227 requested a full tablet report, sending...
I20260929 17:48:37.562170 13070 ts_manager.cc:194] Registered new tserver with Master: f315be7dae4a461db76a509f1e507689 (127.3.183.195:38577)
I20260929 17:48:37.562704 13070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.195:50247
I20260929 17:48:37.566381 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:48:37.569008 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13130
W20260929 17:48:37.569064 13109 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:37.568031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.3.183.254:33227)
0929 17:48:37.568166 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:37.568170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:37.568334 (+ 164us) server_negotiation.cc:407] Connection header received
0929 17:48:37.568417 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:37.568419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:37.568457 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:37.568575 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:37.568937 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38352: BlockingRecv error: recv got EOF from 127.0.0.1:38352 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:48:37.574364 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13261
I20260929 17:48:37.579249 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13392
I20260929 17:48:37.583747 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13039
2026-09-29T17:48:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T17:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:37Z Disabled control of system clock
I20260929 17:48:37.610587 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:45587
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 17:48:37.718724 13531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:37.719123 13531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:37.719192 13531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:37.724965 13531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:37.725045 13531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:37.725064 13531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:37.725085 13531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:37.725117 13531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:37.730191 13531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:45587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:37.731269 13531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:37.732758 13531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:37.741282 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:37.741277 13537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:37.741272 13539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.741328 13531 server_base.cc:1034] running on GCE node
I20260929 17:48:37.742094 13531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:37.743178 13531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:37.744342 13531 hybrid_clock.cc:648] HybridClock initialized: now 1790704117744329 us; error 27 us; skew 500 ppm
I20260929 17:48:37.746157 13531 webserver.cc:460] Webserver started at http://127.3.183.254:42987/ using document root <none> and password file <none>
I20260929 17:48:37.746712 13531 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:37.746773 13531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:37.746966 13531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:37.749027 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "1512f11723924567aaec4a76c3bc5727"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "4ad106b47b5b78c58304a12f61c9751a"
server_key_iv: "236077672cc225a84b5003f3b5c74e07"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.749564 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "1512f11723924567aaec4a76c3bc5727"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "4ad106b47b5b78c58304a12f61c9751a"
server_key_iv: "236077672cc225a84b5003f3b5c74e07"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.753100 13531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:37.755363 13545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.756590 13531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:48:37.756690 13531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "1512f11723924567aaec4a76c3bc5727"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "4ad106b47b5b78c58304a12f61c9751a"
server_key_iv: "236077672cc225a84b5003f3b5c74e07"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.756788 13531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:37.761312 13531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:37.761879 13531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:37.762027 13531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:37.768455 13531 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:45587
I20260929 17:48:37.768522 13597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:45587 every 8 connection(s)
I20260929 17:48:37.769443 13531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:37.772590 13598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:37.775913 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13531
I20260929 17:48:37.776010 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:37.776278 3807 external_mini_cluster.cc:1428] Setting key 60fb2c9e517152efa92e8b054be35f30
I20260929 17:48:37.778312 13598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727: Bootstrap starting.
I20260929 17:48:37.780738 13598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:37.781539 13598 log.cc:824] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:37.781814 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.777703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.3.183.254:45587)
0929 17:48:37.778323 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:37.778339 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:37.778375 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:48:37.779039 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:37.779059 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:37.779412 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:37.779701 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:37.780487 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:37.780491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:37.780503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:37.780512 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:37.780607 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:37.780624 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:37.780706 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:37.780862 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 17:48:37.781083 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260929 17:48:37.783640 13598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727: No bootstrap required, opened a new log
I20260929 17:48:37.787122 13598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512f11723924567aaec4a76c3bc5727" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45587 } }
I20260929 17:48:37.787356 13598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:37.787427 13598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1512f11723924567aaec4a76c3bc5727, State: Initialized, Role: FOLLOWER
I20260929 17:48:37.787907 13598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512f11723924567aaec4a76c3bc5727" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45587 } }
I20260929 17:48:37.788069 13598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:37.788137 13598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:37.788236 13598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:37.789112 13598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512f11723924567aaec4a76c3bc5727" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45587 } }
I20260929 17:48:37.789412 13598 leader_election.cc:302] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1512f11723924567aaec4a76c3bc5727; no voters:
I20260929 17:48:37.789656 13598 leader_election.cc:288] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:37.789752 13603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:37.789959 13603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [term 1 LEADER]: Becoming Leader. State: Replica: 1512f11723924567aaec4a76c3bc5727, State: Running, Role: LEADER
I20260929 17:48:37.790248 13603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512f11723924567aaec4a76c3bc5727" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45587 } }
I20260929 17:48:37.790805 13598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:37.791797 13604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1512f11723924567aaec4a76c3bc5727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512f11723924567aaec4a76c3bc5727" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45587 } } }
I20260929 17:48:37.791945 13604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:37.791819 13605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1512f11723924567aaec4a76c3bc5727. Latest consensus state: current_term: 1 leader_uuid: "1512f11723924567aaec4a76c3bc5727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1512f11723924567aaec4a76c3bc5727" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45587 } } }
I20260929 17:48:37.792160 13605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727 [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:37.792361 13612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:37.795432 13612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:37.801178 13612 catalog_manager.cc:1347] Generated new cluster ID: f5915b58b9374bda9772061f547d644f
I20260929 17:48:37.801242 13612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:37.821166 13612 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:37.821525 13612 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:37.826833 13612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1512f11723924567aaec4a76c3bc5727: Generated new TSK 0
I20260929 17:48:37.827484 13612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:48:37.836303 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45587
--builtin_ntp_servers=127.3.183.212:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 17:48:37.945246 13622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:37.945519 13622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:37.945559 13622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:37.949623 13622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:37.949680 13622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:37.949738 13622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:37.954535 13622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:37.955677 13622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:37.957108 13622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:37.967063 13628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.967111 13622 server_base.cc:1034] running on GCE node
W20260929 17:48:37.967046 13630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:37.967057 13627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:37.967701 13622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:37.968374 13622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:37.969535 13622 hybrid_clock.cc:648] HybridClock initialized: now 1790704117969517 us; error 35 us; skew 500 ppm
I20260929 17:48:37.971351 13622 webserver.cc:460] Webserver started at http://127.3.183.193:33839/ using document root <none> and password file <none>
I20260929 17:48:37.971964 13622 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:37.972034 13622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:37.972249 13622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:37.974346 13622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "53951b9430764b8ca4806e3072d2f666"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "29616ef27fbcfd6d920341b8d353000a"
server_key_iv: "18ddeb0d4ee1fd7193b47bb95c7ec368"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.974907 13622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "53951b9430764b8ca4806e3072d2f666"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "29616ef27fbcfd6d920341b8d353000a"
server_key_iv: "18ddeb0d4ee1fd7193b47bb95c7ec368"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.978591 13622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 17:48:37.981056 13636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.982232 13622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:48:37.982368 13622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "53951b9430764b8ca4806e3072d2f666"
format_stamp: "Formatted at 2026-09-29 17:48:37 on dist-test-slave-wbpt"
server_key: "29616ef27fbcfd6d920341b8d353000a"
server_key_iv: "18ddeb0d4ee1fd7193b47bb95c7ec368"
server_key_version: "encryptionkey@0"
I20260929 17:48:37.982465 13622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:37.986896 13622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:37.987488 13622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:37.987643 13622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:37.988191 13622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:37.989039 13622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:37.989111 13622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.989162 13622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:37.989192 13622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:37.997176 13622 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:34707
I20260929 17:48:37.997223 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:34707 every 8 connection(s)
I20260929 17:48:37.998015 13622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:38.002481 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13622
I20260929 17:48:38.002583 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:38.002858 3807 external_mini_cluster.cc:1428] Setting key 034b44d85596d747b8296b92f9792a20
I20260929 17:48:38.004230 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:37.999912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:41807 (local address 127.3.183.254:45587)
0929 17:48:38.000071 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:38.000075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:38.001104 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 17:48:38.002388 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:38.002392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:38.002436 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:38.002513 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:38.003713 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:38.003717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:38.003719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:38.003730 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:38.003754 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:38.003764 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:38.003877 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:38.004056 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 17:48:38.004103 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:48:38.004953 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45587
--builtin_ntp_servers=127.3.183.212:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 17:48:38.005292 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:38.000288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45587 (local address 127.3.183.193:41807)
0929 17:48:38.000946 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:38.000987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:38.002186 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:38.002705 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:38.002712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:38.002800 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:38.003380 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:38.003382 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:38.003437 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:38.003572 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:38.003881 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:38.003889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:38.003894 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:48:38.004082 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 17:48:38.004314 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20260929 17:48:38.006387 13750 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45587
I20260929 17:48:38.006594 13750 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:38.006960 13750 heartbeater.cc:511] Master 127.3.183.254:45587 requested a full tablet report, sending...
I20260929 17:48:38.008401 13562 ts_manager.cc:194] Registered new tserver with Master: 53951b9430764b8ca4806e3072d2f666 (127.3.183.193:34707)
W20260929 17:48:38.112567 13753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:38.112811 13753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:38.112866 13753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:38.116806 13753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:38.116855 13753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:38.116927 13753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:38.121567 13753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:38.122548 13753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:38.123957 13753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:38.132985 13758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:38.133100 13761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:38.133164 13753 server_base.cc:1034] running on GCE node
W20260929 17:48:38.133005 13759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:38.133713 13753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:38.134346 13753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:38.135504 13753 hybrid_clock.cc:648] HybridClock initialized: now 1790704118135487 us; error 33 us; skew 500 ppm
I20260929 17:48:38.142684 13753 webserver.cc:460] Webserver started at http://127.3.183.194:37225/ using document root <none> and password file <none>
I20260929 17:48:38.143313 13753 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:38.143383 13753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:38.143585 13753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:38.145836 13753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "2f2ea95180ae450fb8b1752042b1d9d1"
format_stamp: "Formatted at 2026-09-29 17:48:38 on dist-test-slave-wbpt"
server_key: "438349141fe40d863b76760818bd5598"
server_key_iv: "bab915d17b7684efe980336757bc1564"
server_key_version: "encryptionkey@0"
I20260929 17:48:38.146425 13753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "2f2ea95180ae450fb8b1752042b1d9d1"
format_stamp: "Formatted at 2026-09-29 17:48:38 on dist-test-slave-wbpt"
server_key: "438349141fe40d863b76760818bd5598"
server_key_iv: "bab915d17b7684efe980336757bc1564"
server_key_version: "encryptionkey@0"
I20260929 17:48:38.150199 13753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:48:38.152909 13767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:38.154204 13753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:38.154318 13753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "2f2ea95180ae450fb8b1752042b1d9d1"
format_stamp: "Formatted at 2026-09-29 17:48:38 on dist-test-slave-wbpt"
server_key: "438349141fe40d863b76760818bd5598"
server_key_iv: "bab915d17b7684efe980336757bc1564"
server_key_version: "encryptionkey@0"
I20260929 17:48:38.154428 13753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:38.159128 13753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:38.159721 13753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:38.159904 13753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:38.160450 13753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:38.161325 13753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:38.161371 13753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:38.161437 13753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:38.161460 13753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:38.170168 13753 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:33937
I20260929 17:48:38.170200 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:33937 every 8 connection(s)
I20260929 17:48:38.171058 13753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:38.171312 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13753
I20260929 17:48:38.171423 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:38.171731 3807 external_mini_cluster.cc:1428] Setting key 69a9633e35ce27ac115c5c2232977fb2
I20260929 17:48:38.173795 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45587
--builtin_ntp_servers=127.3.183.212:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 17:48:38.177434 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:38.172901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:33549 (local address 127.3.183.254:45587)
0929 17:48:38.173039 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:38.173043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:38.174330 (+ 1287us) server_negotiation.cc:407] Connection header received
0929 17:48:38.175619 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:38.175622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:38.175668 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:38.175741 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:38.176953 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:38.176957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:38.176959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:38.176972 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:38.176997 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:38.177005 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:38.177089 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:38.177237 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 17:48:38.177282 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:48:38.178294 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:38.173405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45587 (local address 127.3.183.194:33549)
0929 17:48:38.174186 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:38.174228 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:38.175405 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:38.175959 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:38.175967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:38.176081 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:38.176677 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:38.176679 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:38.176719 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:38.176789 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:38.177081 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:38.177089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:38.177095 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:48:38.177271 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 17:48:38.177518 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":144,"threads_started":1}
I20260929 17:48:38.179510 13881 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45587
I20260929 17:48:38.179730 13881 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:38.180171 13881 heartbeater.cc:511] Master 127.3.183.254:45587 requested a full tablet report, sending...
I20260929 17:48:38.181183 13562 ts_manager.cc:194] Registered new tserver with Master: 2f2ea95180ae450fb8b1752042b1d9d1 (127.3.183.194:33937)
W20260929 17:48:38.297550 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:38.297803 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:38.297856 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:38.301797 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:38.301849 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:38.301923 13884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:38.306660 13884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:38.307626 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:38.308956 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:38.319926 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:38.319924 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:38.320196 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:38.320477 13884 server_base.cc:1034] running on GCE node
I20260929 17:48:38.321033 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:38.321789 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:38.322968 13884 hybrid_clock.cc:648] HybridClock initialized: now 1790704118322934 us; error 47 us; skew 500 ppm
I20260929 17:48:38.325301 13884 webserver.cc:460] Webserver started at http://127.3.183.195:35611/ using document root <none> and password file <none>
I20260929 17:48:38.325953 13884 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:38.326033 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:38.326267 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:38.328575 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "0223c516e2be4568b7eb13b47337dbc2"
format_stamp: "Formatted at 2026-09-29 17:48:38 on dist-test-slave-wbpt"
server_key: "5a065c4f1ec51ca5777418679355f516"
server_key_iv: "c93468653902f225a37356973b7aac71"
server_key_version: "encryptionkey@0"
I20260929 17:48:38.329108 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "0223c516e2be4568b7eb13b47337dbc2"
format_stamp: "Formatted at 2026-09-29 17:48:38 on dist-test-slave-wbpt"
server_key: "5a065c4f1ec51ca5777418679355f516"
server_key_iv: "c93468653902f225a37356973b7aac71"
server_key_version: "encryptionkey@0"
I20260929 17:48:38.332659 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:38.334990 13898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:38.336301 13884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:38.336403 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "0223c516e2be4568b7eb13b47337dbc2"
format_stamp: "Formatted at 2026-09-29 17:48:38 on dist-test-slave-wbpt"
server_key: "5a065c4f1ec51ca5777418679355f516"
server_key_iv: "c93468653902f225a37356973b7aac71"
server_key_version: "encryptionkey@0"
I20260929 17:48:38.336495 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:38.340758 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:38.341300 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:38.341445 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:38.341948 13884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:38.342770 13884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:38.342815 13884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:38.342862 13884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:38.342891 13884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:38.351051 13884 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:40509
I20260929 17:48:38.351082 14011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:40509 every 8 connection(s)
I20260929 17:48:38.351899 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:38.357396 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:38.353499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:58501 (local address 127.3.183.254:45587)
0929 17:48:38.353665 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:38.353669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:38.354659 (+ 990us) server_negotiation.cc:407] Connection header received
0929 17:48:38.355865 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:38.355868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:38.355911 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:38.355986 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:38.356940 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:38.356944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:38.356946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:48:38.356958 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:38.356977 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:48:38.356985 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:38.357057 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:38.357226 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 17:48:38.357270 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:48:38.358177 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:38.353898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45587 (local address 127.3.183.195:58501)
0929 17:48:38.354506 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:38.354543 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:38.355663 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:38.356150 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:38.356157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:38.356239 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:38.356699 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:48:38.356700 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:38.356737 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:48:38.356799 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:38.357053 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:38.357061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:38.357067 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:48:38.357245 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 17:48:38.357465 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260929 17:48:38.359196 14012 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45587
I20260929 17:48:38.359366 14012 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:38.359704 14012 heartbeater.cc:511] Master 127.3.183.254:45587 requested a full tablet report, sending...
I20260929 17:48:38.359841 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 13884
I20260929 17:48:38.359933 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:38.360198 3807 external_mini_cluster.cc:1428] Setting key 702c766534ef368f5d5e324db97fdf3c
I20260929 17:48:38.360747 13562 ts_manager.cc:194] Registered new tserver with Master: 0223c516e2be4568b7eb13b47337dbc2 (127.3.183.195:40509)
I20260929 17:48:38.362082 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:48:38.364892 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13622
W20260929 17:48:38.365541 13601 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:48:38.363866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.3.183.254:45587)
0929 17:48:38.363995 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:38.363999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:38.364214 (+ 215us) server_negotiation.cc:407] Connection header received
0929 17:48:38.364294 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:38.364296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:38.364324 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:38.364383 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:38.365453 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54214: BlockingRecv error: recv got EOF from 127.0.0.1:54214 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:48:38.369949 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13753
I20260929 17:48:38.374518 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13884
I20260929 17:48:38.378688 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 13531
2026-09-29T17:48:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (794 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:38 dist-test-slave-wbpt krb5kdc[14022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:38 dist-test-slave-wbpt krb5kdc[14022](info): set up 2 sockets
Sep 29 17:48:38 dist-test-slave-wbpt krb5kdc[14022](info): commencing operation
krb5kdc: starting...
W20260929 17:48:39.332407 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:39 dist-test-slave-wbpt krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:48:39.451546 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35855
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:35855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:39.557969 14035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:39.558254 14035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:39.558308 14035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:39.562340 14035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:48:39.562402 14035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:39.562429 14035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:39.562448 14035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:48:39.562469 14035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:48:39.567710 14035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:35855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:39.568804 14035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:39.569820 14035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:39.578364 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:39.578424 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:39.578490 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:39.578516 14035 server_base.cc:1034] running on GCE node
I20260929 17:48:39.579180 14035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:39.580343 14035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:39.581517 14035 hybrid_clock.cc:648] HybridClock initialized: now 1790704119581493 us; error 39 us; skew 500 ppm
Sep 29 17:48:39 dist-test-slave-wbpt krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704119, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:39.584524 14035 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:48:39.585453 14035 webserver.cc:460] Webserver started at http://127.3.183.254:40453/ using document root <none> and password file <none>
I20260929 17:48:39.585990 14035 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:39.586057 14035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:39.586233 14035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:39.588615 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "07f4bee42039469eafb367452bbfe2ce"
format_stamp: "Formatted at 2026-09-29 17:48:39 on dist-test-slave-wbpt"
server_key: "a0d2bea705fd84950154be18c02b1f4e"
server_key_iv: "858698abc9bb35b3042c349d16b0e3de"
server_key_version: "encryptionkey@0"
I20260929 17:48:39.589149 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "07f4bee42039469eafb367452bbfe2ce"
format_stamp: "Formatted at 2026-09-29 17:48:39 on dist-test-slave-wbpt"
server_key: "a0d2bea705fd84950154be18c02b1f4e"
server_key_iv: "858698abc9bb35b3042c349d16b0e3de"
server_key_version: "encryptionkey@0"
I20260929 17:48:39.592947 14035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:48:39.595402 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:39.596731 14035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:39.596848 14035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "07f4bee42039469eafb367452bbfe2ce"
format_stamp: "Formatted at 2026-09-29 17:48:39 on dist-test-slave-wbpt"
server_key: "a0d2bea705fd84950154be18c02b1f4e"
server_key_iv: "858698abc9bb35b3042c349d16b0e3de"
server_key_version: "encryptionkey@0"
I20260929 17:48:39.596958 14035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:39.623456 14035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:39.626421 14035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:39.626624 14035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:39.633211 14035 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:35855
I20260929 17:48:39.633462 14102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:35855 every 8 connection(s)
I20260929 17:48:39.634187 14035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:48:39.636907 14103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:39.637740 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 14035
I20260929 17:48:39.637846 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:48:39.638167 3807 external_mini_cluster.cc:1428] Setting key 8af8948d2fd7aebf2b7e9432ea013564
I20260929 17:48:39.642597 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce: Bootstrap starting.
I20260929 17:48:39.644773 14103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:39.645540 14103 log.cc:824] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:48:39 dist-test-slave-wbpt krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:39.647396 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce: No bootstrap required, opened a new log
I20260929 17:48:39.650732 14103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f4bee42039469eafb367452bbfe2ce" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35855 } }
I20260929 17:48:39.650941 14103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:39.650996 14103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f4bee42039469eafb367452bbfe2ce, State: Initialized, Role: FOLLOWER
I20260929 17:48:39.651407 14103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f4bee42039469eafb367452bbfe2ce" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35855 } }
I20260929 17:48:39.651554 14103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:48:39.651620 14103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:48:39.651711 14103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:39.652042 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:39.639597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.3.183.254:35855)
0929 17:48:39.640059 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:39.640075 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:39.640112 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:48:39.640770 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:39.640788 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:39.641131 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:39.641438 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:39.642290 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:39.643369 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:39.644306 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:39.644785 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:39.644950 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:39.647466 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:39.647472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:39.647489 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:39.647523 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:39.649159 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:39.649601 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:39.649606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:39.649611 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:39.649728 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:39.650007 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:39.650011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:39.650013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:39.650395 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:39.650600 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:39.650883 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 17:48:39.651201 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":150,"threads_started":1}
I20260929 17:48:39.652593 14103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f4bee42039469eafb367452bbfe2ce" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35855 } }
I20260929 17:48:39.652890 14103 leader_election.cc:302] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07f4bee42039469eafb367452bbfe2ce; no voters:
I20260929 17:48:39.653172 14103 leader_election.cc:288] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:48:39.653250 14108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:39.653414 14108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [term 1 LEADER]: Becoming Leader. State: Replica: 07f4bee42039469eafb367452bbfe2ce, State: Running, Role: LEADER
I20260929 17:48:39.653733 14108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f4bee42039469eafb367452bbfe2ce" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35855 } }
I20260929 17:48:39.654422 14103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:48:39.655342 14109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f4bee42039469eafb367452bbfe2ce. Latest consensus state: current_term: 1 leader_uuid: "07f4bee42039469eafb367452bbfe2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f4bee42039469eafb367452bbfe2ce" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35855 } } }
I20260929 17:48:39.655562 14109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [sys.catalog]: This master's current role is: LEADER
I20260929 17:48:39.655870 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07f4bee42039469eafb367452bbfe2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f4bee42039469eafb367452bbfe2ce" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35855 } } }
I20260929 17:48:39.655972 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce [sys.catalog]: This master's current role is: LEADER
W20260929 17:48:39.658867 14123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:48:39.658932 14123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:48:39.659027 14117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:48:39.659996 14117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:48:39.665688 14117 catalog_manager.cc:1347] Generated new cluster ID: 3e0d0f6b49ed459daa780490529d0571
I20260929 17:48:39.665764 14117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:48:39.690920 14117 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:48:39.692068 14117 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:48:39.697896 14117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce: Generated new TSK 0
I20260929 17:48:39.698592 14117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:48:39.731886 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35855
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:39.842461 14129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:39.842721 14129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:39.842764 14129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:39.846717 14129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:39.846787 14129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:39.846845 14129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:48:39.852135 14129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:39.853242 14129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:39.854265 14129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:39.863946 14134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:39.864056 14129 server_base.cc:1034] running on GCE node
W20260929 17:48:39.863962 14137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:39.863953 14135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:39.864569 14129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:39.865152 14129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:39.866307 14129 hybrid_clock.cc:648] HybridClock initialized: now 1790704119866292 us; error 27 us; skew 500 ppm
Sep 29 17:48:39 dist-test-slave-wbpt krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704119, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:39.868932 14129 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:48:39.869812 14129 webserver.cc:460] Webserver started at http://127.3.183.193:44275/ using document root <none> and password file <none>
I20260929 17:48:39.870334 14129 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:39.870397 14129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:39.870587 14129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:39.872603 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "78202fb5d26848a2889ffbece5df74a6"
format_stamp: "Formatted at 2026-09-29 17:48:39 on dist-test-slave-wbpt"
server_key: "588ef365c8f9ce810dbf79fb9d66363a"
server_key_iv: "8d7dae4fe43cc778a61d71d901b631ff"
server_key_version: "encryptionkey@0"
I20260929 17:48:39.873137 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "78202fb5d26848a2889ffbece5df74a6"
format_stamp: "Formatted at 2026-09-29 17:48:39 on dist-test-slave-wbpt"
server_key: "588ef365c8f9ce810dbf79fb9d66363a"
server_key_iv: "8d7dae4fe43cc778a61d71d901b631ff"
server_key_version: "encryptionkey@0"
I20260929 17:48:39.876748 14129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 17:48:39.879256 14144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:39.880498 14129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:48:39.880601 14129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "78202fb5d26848a2889ffbece5df74a6"
format_stamp: "Formatted at 2026-09-29 17:48:39 on dist-test-slave-wbpt"
server_key: "588ef365c8f9ce810dbf79fb9d66363a"
server_key_iv: "8d7dae4fe43cc778a61d71d901b631ff"
server_key_version: "encryptionkey@0"
I20260929 17:48:39.880695 14129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:39.899585 14129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:39.902702 14129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:39.902875 14129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:39.903430 14129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:39.904237 14129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:39.904291 14129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:39.904345 14129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:39.904372 14129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:39.912313 14129 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:40455
I20260929 17:48:39.912343 14257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:40455 every 8 connection(s)
I20260929 17:48:39.913130 14129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:48:39.918143 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 14129
I20260929 17:48:39.918238 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:48:39.918478 3807 external_mini_cluster.cc:1428] Setting key 72a4d94fe2d3e4ab279553d1b74c1c10
Sep 29 17:48:39 dist-test-slave-wbpt krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704119, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:39.926234 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:39.914809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:55305 (local address 127.3.183.254:35855)
0929 17:48:39.914928 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:39.914931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:39.915769 (+ 838us) server_negotiation.cc:407] Connection header received
0929 17:48:39.917020 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:39.917023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:39.917076 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:39.917158 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:39.918353 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:39.918928 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:39.920250 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:39.920475 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:39.920560 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:39.922861 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:39.922864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:39.922866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:39.922889 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:39.924308 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:39.924785 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:39.924789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:39.924791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:39.924864 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:39.925198 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:39.925201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:39.925203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:39.925372 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:39.925454 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:39.925945 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 17:48:39.926072 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:48:39.927006 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:39.915144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35855 (local address 127.3.183.193:55305)
0929 17:48:39.915603 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:39.915644 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:39.916833 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:39.917374 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:39.917381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:39.917736 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:39.918201 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:39.918219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:39.919047 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:39.919049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:39.920130 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:39.920139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:39.920607 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:39.920611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:39.920761 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:39.921294 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:39.921310 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:39.922710 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:39.924415 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:39.924421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:39.924435 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:39.924688 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:39.924947 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:39.924950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:39.924952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:39.925095 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:39.925473 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:39.925479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:39.925718 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 17:48:39.925882 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 17:48:39.926189 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260929 17:48:39.928151 14258 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35855
I20260929 17:48:39.928364 14258 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:39.928834 14258 heartbeater.cc:511] Master 127.3.183.254:35855 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:48:39.930377 14067 ts_manager.cc:194] Registered new tserver with Master: 78202fb5d26848a2889ffbece5df74a6 (127.3.183.193:40455)
I20260929 17:48:39.931506 14067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:55305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:48:39.942687 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35855
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:40.050808 14263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:40.051084 14263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:40.051127 14263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:40.055280 14263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:40.055361 14263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:40.055438 14263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:48:40.060899 14263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:40.061949 14263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:40.062952 14263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:40.072093 14268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:40.072172 14263 server_base.cc:1034] running on GCE node
W20260929 17:48:40.072129 14269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:40.072135 14271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:40.072692 14263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:40.073297 14263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:40.074466 14263 hybrid_clock.cc:648] HybridClock initialized: now 1790704120074450 us; error 30 us; skew 500 ppm
Sep 29 17:48:40 dist-test-slave-wbpt krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704120, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:40.077131 14263 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:48:40.077991 14263 webserver.cc:460] Webserver started at http://127.3.183.194:46793/ using document root <none> and password file <none>
I20260929 17:48:40.078490 14263 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:40.078552 14263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:40.078778 14263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:40.080760 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "519b993bd7c64192b74ef24ca50fbafb"
format_stamp: "Formatted at 2026-09-29 17:48:40 on dist-test-slave-wbpt"
server_key: "7e080655cf412c92e9da3b1233d24dda"
server_key_iv: "75d5f75b275f7a96a1dc4e150d391f74"
server_key_version: "encryptionkey@0"
I20260929 17:48:40.081266 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "519b993bd7c64192b74ef24ca50fbafb"
format_stamp: "Formatted at 2026-09-29 17:48:40 on dist-test-slave-wbpt"
server_key: "7e080655cf412c92e9da3b1233d24dda"
server_key_iv: "75d5f75b275f7a96a1dc4e150d391f74"
server_key_version: "encryptionkey@0"
I20260929 17:48:40.084860 14263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:48:40.087250 14278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:40.088594 14263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:48:40.088701 14263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "519b993bd7c64192b74ef24ca50fbafb"
format_stamp: "Formatted at 2026-09-29 17:48:40 on dist-test-slave-wbpt"
server_key: "7e080655cf412c92e9da3b1233d24dda"
server_key_iv: "75d5f75b275f7a96a1dc4e150d391f74"
server_key_version: "encryptionkey@0"
I20260929 17:48:40.088796 14263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:40.102878 14263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:40.105877 14263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:40.106051 14263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:40.106617 14263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:40.107388 14263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:40.107445 14263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:40.107496 14263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:40.107530 14263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:40.115891 14263 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:40979
I20260929 17:48:40.115921 14391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:40979 every 8 connection(s)
I20260929 17:48:40.116752 14263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:48:40.118603 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 14263
I20260929 17:48:40.118716 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:48:40.119012 3807 external_mini_cluster.cc:1428] Setting key 54222c7fe56b06b8c3f0113819f867f0
Sep 29 17:48:40 dist-test-slave-wbpt krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704120, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:48:40.130321 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.118628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:53777 (local address 127.3.183.254:35855)
0929 17:48:40.118778 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.118784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.119793 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 17:48:40.121174 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.121177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.121226 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.121297 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:40.122444 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.122976 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.124141 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.124355 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.124441 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.126939 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:40.126942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:40.126944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:40.126967 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:40.128326 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:40.128790 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:40.128794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:40.128796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:40.128871 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:40.129179 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:40.129182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:40.129184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:40.129357 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:40.129440 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.130031 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.130160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:48:40.131072 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.119105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35855 (local address 127.3.183.194:53777)
0929 17:48:40.119638 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.119683 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:40.120952 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:40.121481 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:40.121487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:40.121832 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:40.122305 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.122319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.123097 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.123100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.124028 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.124039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.124489 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.124493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.124670 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.125250 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:40.125265 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:40.126780 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:40.128427 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:40.128432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:40.128446 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:40.128695 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:40.128954 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:40.128956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:40.128959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:40.129091 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:40.129454 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:40.129461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:40.129769 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 17:48:40.129969 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 17:48:40.130235 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":139,"threads_started":1}
I20260929 17:48:40.132261 14392 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35855
I20260929 17:48:40.132479 14392 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:40.132926 14392 heartbeater.cc:511] Master 127.3.183.254:35855 requested a full tablet report, sending...
I20260929 17:48:40.133906 14067 ts_manager.cc:194] Registered new tserver with Master: 519b993bd7c64192b74ef24ca50fbafb (127.3.183.194:40979)
I20260929 17:48:40.134455 14067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:53777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:48:40.146867 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35855
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:48:40.255677 14397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:48:40.255985 14397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:48:40.256078 14397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:48:40.260068 14397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:48:40.260118 14397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:48:40.260174 14397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:48:40.265343 14397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:48:40.266285 14397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:48:40.267210 14397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:48:40.276518 14402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:40.276469 14405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:48:40.276485 14403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:48:40.276715 14397 server_base.cc:1034] running on GCE node
I20260929 17:48:40.277163 14397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:48:40.277726 14397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:48:40.278872 14397 hybrid_clock.cc:648] HybridClock initialized: now 1790704120278857 us; error 24 us; skew 500 ppm
Sep 29 17:48:40 dist-test-slave-wbpt krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704120, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:48:40.281615 14397 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:48:40.282480 14397 webserver.cc:460] Webserver started at http://127.3.183.195:40675/ using document root <none> and password file <none>
I20260929 17:48:40.282992 14397 fs_manager.cc:359] Metadata directory not provided
I20260929 17:48:40.283079 14397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:48:40.283259 14397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:48:40.285311 14397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "630ca74597224efa90e8def374009eb4"
format_stamp: "Formatted at 2026-09-29 17:48:40 on dist-test-slave-wbpt"
server_key: "16c131f37082318b5915b7385d0ff481"
server_key_iv: "a7ab265e86225406e7fc5702ee02c856"
server_key_version: "encryptionkey@0"
I20260929 17:48:40.285818 14397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "630ca74597224efa90e8def374009eb4"
format_stamp: "Formatted at 2026-09-29 17:48:40 on dist-test-slave-wbpt"
server_key: "16c131f37082318b5915b7385d0ff481"
server_key_iv: "a7ab265e86225406e7fc5702ee02c856"
server_key_version: "encryptionkey@0"
I20260929 17:48:40.289271 14397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 17:48:40.291546 14412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:40.292876 14397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260929 17:48:40.292976 14397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "630ca74597224efa90e8def374009eb4"
format_stamp: "Formatted at 2026-09-29 17:48:40 on dist-test-slave-wbpt"
server_key: "16c131f37082318b5915b7385d0ff481"
server_key_iv: "a7ab265e86225406e7fc5702ee02c856"
server_key_version: "encryptionkey@0"
I20260929 17:48:40.293098 14397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:48:40.303128 14397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:48:40.305757 14397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:48:40.305914 14397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:48:40.306423 14397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:48:40.307173 14397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:48:40.307219 14397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:40.307267 14397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:48:40.307296 14397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:48:40.316529 14397 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:32883
I20260929 17:48:40.316906 14525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:32883 every 8 connection(s)
I20260929 17:48:40.317531 14397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:48:40.322822 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 14397
I20260929 17:48:40.322938 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:48:40.323210 3807 external_mini_cluster.cc:1428] Setting key 3ceb1bd95aa81ba1733f9d127725deab
Sep 29 17:48:40 dist-test-slave-wbpt krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704120, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:48:40.330482 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.319363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:55167 (local address 127.3.183.254:35855)
0929 17:48:40.319489 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.319493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.320334 (+ 841us) server_negotiation.cc:407] Connection header received
0929 17:48:40.321637 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.321640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.321695 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.321765 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:40.323000 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.323630 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.325048 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.325308 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.325388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.327555 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:40.327558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:40.327560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:40.327585 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:40.328861 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:40.329290 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:40.329293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:40.329294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:40.329363 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:40.329623 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:40.329625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:40.329626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:40.329786 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:40.329853 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.330271 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.330362 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:48:40.331318 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.319715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35855 (local address 127.3.183.195:55167)
0929 17:48:40.320199 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.320239 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:40.321429 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:40.321928 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:40.321935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:40.322278 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:48:40.322825 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.322846 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.323754 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.323757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.324909 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.324918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.325433 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.325436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.325582 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.326081 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:48:40.326097 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:48:40.327412 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:48:40.328959 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:40.328963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:40.328974 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:40.329214 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:40.329428 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:48:40.329430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:48:40.329431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:48:40.329559 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:48:40.329860 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:48:40.329865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:48:40.330083 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 17:48:40.330233 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 17:48:40.330472 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":118,"threads_started":1}
I20260929 17:48:40.332384 14526 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35855
I20260929 17:48:40.332625 14526 heartbeater.cc:464] Registering TS with master...
I20260929 17:48:40.333091 14526 heartbeater.cc:511] Master 127.3.183.254:35855 requested a full tablet report, sending...
I20260929 17:48:40.334053 14067 ts_manager.cc:194] Registered new tserver with Master: 630ca74597224efa90e8def374009eb4 (127.3.183.195:32883)
I20260929 17:48:40.334589 14067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:55167
I20260929 17:48:40.336792 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:48:40.345398 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.338559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.3.183.254:35855)
0929 17:48:40.338654 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.338658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.338882 (+ 224us) server_negotiation.cc:407] Connection header received
0929 17:48:40.338983 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.338986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.339045 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.339105 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:48:40.339771 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.340312 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.341131 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.341346 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.341430 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.342716 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:48:40.342720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:48:40.342723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:48:40.342747 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:48:40.343804 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:40.344226 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:40.344229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:40.344231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:40.344308 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:48:40.344560 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:48:40.344563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:48:40.344565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:48:40.344777 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:48:40.344898 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.345136 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.345244 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 17:48:40.348306 14067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:48:40.351226 14067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:48:40.365805 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.360040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.3.183.193:40455)
0929 17:48:40.360454 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.360473 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.361033 (+ 560us) server_negotiation.cc:407] Connection header received
0929 17:48:40.361128 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.361134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.361364 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.361518 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:40.361988 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.362882 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.364484 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.365174 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.365270 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.365444 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.365564 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.365656 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":151,"threads_started":1}
I20260929 17:48:40.365799 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.359909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:40455 (local address 127.0.0.1:52584)
0929 17:48:40.360681 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.360734 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:40.360871 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:40.361701 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:40.361704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:40.361729 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:40.361864 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.361869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.363012 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.363016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.364373 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.364385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.365295 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.365298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.365401 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.365438 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:48:40.365539 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 17:48:40.365653 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":44,"spinlock_wait_cycles":14208,"thread_start_us":88,"threads_started":1}
I20260929 17:48:40.366295 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.360069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:32883 (local address 127.0.0.1:52822)
0929 17:48:40.360702 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.360748 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:40.360870 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:40.361642 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:40.361648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:40.361678 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:40.361842 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.361850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.363388 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.363400 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.364596 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.364604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.365690 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.365694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.365782 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.365806 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:48:40.365859 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 17:48:40.366147 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":60,"spinlock_wait_cycles":4864,"thread_start_us":119,"threads_started":1}
I20260929 17:48:40.366591 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.360453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.3.183.195:32883)
0929 17:48:40.361184 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.361195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.361215 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:48:40.361285 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.361290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.361479 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.362003 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:40.362144 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.363242 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.364719 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.365536 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.366220 (+ 684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.366323 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.366363 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.366459 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":97,"threads_started":1}
I20260929 17:48:40.367103 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.361130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40979 (local address 127.0.0.1:49524)
0929 17:48:40.361586 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.361599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:40.361690 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:40.362255 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:40.362258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:40.362279 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:40.362361 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.362367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.363613 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.363616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.364714 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.364724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.365913 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.365917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.365998 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.366792 (+ 794us) client_negotiation.cc:769] Sending connection context
0929 17:48:40.366882 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 17:48:40.366959 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":3,"spinlock_wait_cycles":4352,"thread_start_us":97,"threads_started":1}
I20260929 17:48:40.367950 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.361276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.3.183.194:40979)
0929 17:48:40.361853 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.361864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.361885 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:48:40.361951 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.361956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.362115 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.362250 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:40.362461 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.363491 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.364837 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.365610 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.365707 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.365853 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.367682 (+ 1829us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.367817 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":129,"threads_started":1}
I20260929 17:48:40.369036 14192 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7cb6a0a7094da3abc43dd1dac00752 (DEFAULT_TABLE table=test-table [id=bcac2680dcf14598afe85fdb8344f9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:40.369174 14460 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7cb6a0a7094da3abc43dd1dac00752 (DEFAULT_TABLE table=test-table [id=bcac2680dcf14598afe85fdb8344f9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:40.370019 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7cb6a0a7094da3abc43dd1dac00752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:40.370203 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7cb6a0a7094da3abc43dd1dac00752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:40.370442 14326 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7cb6a0a7094da3abc43dd1dac00752 (DEFAULT_TABLE table=test-table [id=bcac2680dcf14598afe85fdb8344f9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:48:40.371426 14326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7cb6a0a7094da3abc43dd1dac00752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:48:40.376629 14543 tablet_bootstrap.cc:492] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: Bootstrap starting.
I20260929 17:48:40.377384 14544 tablet_bootstrap.cc:492] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Bootstrap starting.
I20260929 17:48:40.378928 14543 tablet_bootstrap.cc:654] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:40.379243 14545 tablet_bootstrap.cc:492] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: Bootstrap starting.
I20260929 17:48:40.379325 14544 tablet_bootstrap.cc:654] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:40.380227 14543 log.cc:824] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:40.380856 14544 log.cc:824] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:40.381860 14545 tablet_bootstrap.cc:654] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: Neither blocks nor log segments found. Creating new log.
I20260929 17:48:40.382203 14543 tablet_bootstrap.cc:492] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: No bootstrap required, opened a new log
I20260929 17:48:40.382395 14543 ts_tablet_manager.cc:1397] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:48:40.382658 14545 log.cc:824] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: Log is configured to *not* fsync() on all Append() calls
I20260929 17:48:40.383121 14544 tablet_bootstrap.cc:492] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: No bootstrap required, opened a new log
I20260929 17:48:40.383309 14544 ts_tablet_manager.cc:1397] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:48:40.384680 14545 tablet_bootstrap.cc:492] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: No bootstrap required, opened a new log
I20260929 17:48:40.384915 14545 ts_tablet_manager.cc:1397] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:48:40.385594 14543 raft_consensus.cc:348] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.385799 14543 raft_consensus.cc:374] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:40.385849 14543 raft_consensus.cc:729] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78202fb5d26848a2889ffbece5df74a6, State: Initialized, Role: FOLLOWER
I20260929 17:48:40.386248 14543 consensus_queue.cc:260] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.386875 14258 heartbeater.cc:503] Master 127.3.183.254:35855 was elected leader, sending a full tablet report...
I20260929 17:48:40.387217 14543 ts_tablet_manager.cc:1428] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:48:40.387315 14544 raft_consensus.cc:348] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.387568 14544 raft_consensus.cc:374] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:40.387619 14544 raft_consensus.cc:729] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630ca74597224efa90e8def374009eb4, State: Initialized, Role: FOLLOWER
I20260929 17:48:40.388093 14544 consensus_queue.cc:260] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.388811 14526 heartbeater.cc:503] Master 127.3.183.254:35855 was elected leader, sending a full tablet report...
I20260929 17:48:40.388664 14545 raft_consensus.cc:348] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.388911 14545 raft_consensus.cc:374] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:48:40.388965 14545 raft_consensus.cc:729] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519b993bd7c64192b74ef24ca50fbafb, State: Initialized, Role: FOLLOWER
I20260929 17:48:40.389346 14545 consensus_queue.cc:260] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.390144 14545 ts_tablet_manager.cc:1428] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:48:40.390144 14544 ts_tablet_manager.cc:1428] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:48:40.390452 14392 heartbeater.cc:503] Master 127.3.183.254:35855 was elected leader, sending a full tablet report...
W20260929 17:48:40.414227 14259 tablet.cc:2369] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:48:40.568526 14527 tablet.cc:2369] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:48:40.617870 14393 tablet.cc:2369] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:48:40.646371 14550 raft_consensus.cc:482] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:48:40.646574 14550 raft_consensus.cc:504] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.647614 14550 leader_election.cc:288] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78202fb5d26848a2889ffbece5df74a6 (127.3.183.193:40455), 519b993bd7c64192b74ef24ca50fbafb (127.3.183.194:40979)
I20260929 17:48:40.651546 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.647899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:40455 (local address 127.3.183.195:36231)
0929 17:48:40.648303 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.648314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:40.648480 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:40.648766 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:40.648770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:40.648790 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:40.648911 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.648916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.649744 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.649748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.650511 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.650520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.651132 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.651135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.651238 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.651260 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:48:40.651322 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:48:40.651396 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":71,"spinlock_wait_cycles":34304,"thread_start_us":98,"threads_started":1}
I20260929 17:48:40.651533 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.647972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:36231 (local address 127.3.183.193:40455)
0929 17:48:40.648102 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.648107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.648384 (+ 277us) server_negotiation.cc:407] Connection header received
0929 17:48:40.648595 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.648598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.648653 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.648741 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:40.649025 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.649646 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.650643 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.651028 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.651128 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.651182 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.651348 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.651412 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:48:40.651757 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.647973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:53971 (local address 127.3.183.194:40979)
0929 17:48:40.648102 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.648107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.648384 (+ 277us) server_negotiation.cc:407] Connection header received
0929 17:48:40.648868 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.648872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.648918 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.648988 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:48:40.649351 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.649886 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.650873 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.651252 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.651353 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.651400 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.651565 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.651632 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:48:40.651912 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.647900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40979 (local address 127.3.183.195:53971)
0929 17:48:40.648298 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.648314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:48:40.648480 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:48:40.649121 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:48:40.649124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:48:40.649141 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:48:40.649246 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.649252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.649991 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.649994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.650754 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:48:40.650762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.651358 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.651361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:48:40.651453 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.651471 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 17:48:40.651532 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:48:40.651720 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"spinlock_wait_cycles":92800}
I20260929 17:48:40.652247 14212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7cb6a0a7094da3abc43dd1dac00752" candidate_uuid: "630ca74597224efa90e8def374009eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78202fb5d26848a2889ffbece5df74a6" is_pre_election: true
I20260929 17:48:40.652541 14212 raft_consensus.cc:2454] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630ca74597224efa90e8def374009eb4 in term 0.
I20260929 17:48:40.652531 14346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7cb6a0a7094da3abc43dd1dac00752" candidate_uuid: "630ca74597224efa90e8def374009eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519b993bd7c64192b74ef24ca50fbafb" is_pre_election: true
I20260929 17:48:40.652781 14346 raft_consensus.cc:2454] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630ca74597224efa90e8def374009eb4 in term 0.
I20260929 17:48:40.652982 14416 leader_election.cc:302] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630ca74597224efa90e8def374009eb4, 78202fb5d26848a2889ffbece5df74a6; no voters:
I20260929 17:48:40.653263 14550 raft_consensus.cc:2781] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:48:40.653332 14550 raft_consensus.cc:482] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:48:40.653362 14550 raft_consensus.cc:3037] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:40.654376 14550 raft_consensus.cc:504] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.654776 14550 leader_election.cc:288] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [CANDIDATE]: Term 1 election: Requested vote from peers 78202fb5d26848a2889ffbece5df74a6 (127.3.183.193:40455), 519b993bd7c64192b74ef24ca50fbafb (127.3.183.194:40979)
I20260929 17:48:40.655170 14212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7cb6a0a7094da3abc43dd1dac00752" candidate_uuid: "630ca74597224efa90e8def374009eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78202fb5d26848a2889ffbece5df74a6"
I20260929 17:48:40.655254 14346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7cb6a0a7094da3abc43dd1dac00752" candidate_uuid: "630ca74597224efa90e8def374009eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519b993bd7c64192b74ef24ca50fbafb"
I20260929 17:48:40.655304 14212 raft_consensus.cc:3037] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:40.655404 14346 raft_consensus.cc:3037] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:48:40.656527 14212 raft_consensus.cc:2454] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630ca74597224efa90e8def374009eb4 in term 1.
I20260929 17:48:40.656769 14346 raft_consensus.cc:2454] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630ca74597224efa90e8def374009eb4 in term 1.
I20260929 17:48:40.656797 14416 leader_election.cc:302] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630ca74597224efa90e8def374009eb4, 78202fb5d26848a2889ffbece5df74a6; no voters:
I20260929 17:48:40.656981 14550 raft_consensus.cc:2781] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:48:40.657212 14550 raft_consensus.cc:686] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 630ca74597224efa90e8def374009eb4, State: Running, Role: LEADER
I20260929 17:48:40.657533 14550 consensus_queue.cc:237] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } }
I20260929 17:48:40.660640 14067 catalog_manager.cc:5495] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 630ca74597224efa90e8def374009eb4 (127.3.183.195). New cstate: current_term: 1 leader_uuid: "630ca74597224efa90e8def374009eb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "630ca74597224efa90e8def374009eb4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 } health_report { overall_health: UNKNOWN } } }
I20260929 17:48:40.728675 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.725161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.3.183.195:32883)
0929 17:48:40.725290 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:48:40.725293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:48:40.725307 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:48:40.725385 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:48:40.725388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:48:40.725439 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:48:40.725539 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:48:40.725799 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.726304 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.727215 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:48:40.727453 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:48:40.727544 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:48:40.727914 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:48:40.728340 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:48:40.728439 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 17:48:40.728500 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 17:48:40.728558 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:48:40.734606 14346 raft_consensus.cc:1261] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 1 FOLLOWER]: Refusing update from remote peer 630ca74597224efa90e8def374009eb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:40.734606 14212 raft_consensus.cc:1261] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 1 FOLLOWER]: Refusing update from remote peer 630ca74597224efa90e8def374009eb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:48:40.735424 14553 consensus_queue.cc:1035] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519b993bd7c64192b74ef24ca50fbafb" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:40.735584 14550 consensus_queue.cc:1035] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78202fb5d26848a2889ffbece5df74a6" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 40455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:48:40.744598 14558 mvcc.cc:205] Tried to move back new op lower bound from 7334724078521589760 to 7334724078215790592. Current Snapshot: MvccSnapshot[applied={T|T < 7334724078521589760 or (T in {7334724078521589760})}]
I20260929 17:48:40.751470 14557 mvcc.cc:205] Tried to move back new op lower bound from 7334724078521589760 to 7334724078215790592. Current Snapshot: MvccSnapshot[applied={T|T < 7334724078521589760}]
I20260929 17:48:40.752622 14067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:40.752799 14067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:48:40.755333 14067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07f4bee42039469eafb367452bbfe2ce: Sending DeleteTablet for 3 replicas of tablet 3e7cb6a0a7094da3abc43dd1dac00752
I20260929 17:48:40.756124 14192 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7cb6a0a7094da3abc43dd1dac00752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:40 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:52584
I20260929 17:48:40.756354 14326 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7cb6a0a7094da3abc43dd1dac00752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:40 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:49524
I20260929 17:48:40.756736 14566 tablet_replica.cc:330] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6: stopping tablet replica
I20260929 17:48:40.756685 14460 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7cb6a0a7094da3abc43dd1dac00752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:48:40 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:52822
I20260929 17:48:40.757063 14566 raft_consensus.cc:2229] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:48:40.757136 14568 tablet_replica.cc:330] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: stopping tablet replica
I20260929 17:48:40.757426 14566 raft_consensus.cc:2258] T 3e7cb6a0a7094da3abc43dd1dac00752 P 78202fb5d26848a2889ffbece5df74a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:40.757599 14568 raft_consensus.cc:2229] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:48:40.757829 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 14129
I20260929 17:48:40.758318 14568 raft_consensus.cc:2258] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:40.759799 14568 ts_tablet_manager.cc:1905] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:48:40.762940 14568 ts_tablet_manager.cc:1918] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:48:40.763059 14568 log.cc:1196] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/3e7cb6a0a7094da3abc43dd1dac00752
I20260929 17:48:40.763381 14568 ts_tablet_manager.cc:1939] T 3e7cb6a0a7094da3abc43dd1dac00752 P 630ca74597224efa90e8def374009eb4: Deleting consensus metadata
I20260929 17:48:40.763397 14567 tablet_replica.cc:330] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb: stopping tablet replica
I20260929 17:48:40.763914 14567 raft_consensus.cc:2229] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 17:48:40.764050 14054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:40455 (error 108)
I20260929 17:48:40.764468 14567 raft_consensus.cc:2258] T 3e7cb6a0a7094da3abc43dd1dac00752 P 519b993bd7c64192b74ef24ca50fbafb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:48:40.764708 14053 catalog_manager.cc:4841] TS 630ca74597224efa90e8def374009eb4 (127.3.183.195:32883): tablet 3e7cb6a0a7094da3abc43dd1dac00752 (table test-table [id=bcac2680dcf14598afe85fdb8344f9c1]) successfully deleted
I20260929 17:48:40.764941 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 14263
I20260929 17:48:40.768416 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.768081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:40455 (local address 127.0.0.1:52592)
0929 17:48:40.768239 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.768321 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:40455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 17:48:40.768621 14054 catalog_manager.cc:4584] TS 78202fb5d26848a2889ffbece5df74a6 (127.3.183.193:40455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e7cb6a0a7094da3abc43dd1dac00752: Network error: Client connection negotiation failed: client connection to 127.3.183.193:40455: connect: Connection refused (error 111)
I20260929 17:48:40.770782 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 14397
I20260929 17:48:40.771606 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:48:40.770814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:40979 (local address 127.0.0.1:49540)
0929 17:48:40.771465 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 17:48:40.771522 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574}
I20260929 17:48:40.775604 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 14035
2026-09-29T17:48:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2399 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:48:40 dist-test-slave-wbpt krb5kdc[14572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:48:40 dist-test-slave-wbpt krb5kdc[14572](info): set up 2 sockets
Sep 29 17:48:40 dist-test-slave-wbpt krb5kdc[14572](info): commencing operation
krb5kdc: starting...
W20260929 17:48:41.744620 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:48:41 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:48:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/rangeradmin_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/rangeradmin_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/rangerlookup_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/rangerlookup_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
I20260929 17:48:41.907567 3807 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6qiWXY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/postgres -l logfile start
2026-09-29 17:48:44.547 UTC [14602] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 17:48:44.547 UTC [14602] LOG: listening on IPv4 address "127.3.183.212", port 55089
2026-09-29 17:48:44.552 UTC [14602] LOG: listening on Unix socket "/tmp/.s.PGSQL.55089"
2026-09-29 17:48:44.581 UTC [14605] LOG: database system was shut down at 2026-09-29 17:48:42 UTC
2026-09-29 17:48:44.586 UTC [14602] LOG: database system is ready to accept connections
I20260929 17:48:45.436261 3807 mini_postgres.cc:96] Postgres bound to 55089
2026-09-29 17:48:45.443 UTC [14613] FATAL: database "slave" does not exist
127.3.183.212:55089 - accepting connections
I20260929 17:48:45.443864 3807 mini_ranger.cc:161] Starting Ranger...
I20260929 17:48:45.458359 3807 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 17:48:45.970803 3807 mini_ranger.cc:87] Created ranger Postgres database
I20260929 17:48:45.970896 3807 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-admin
2026-09-29 17:48:46,278 [I] DB FLAVOR :POSTGRES
2026-09-29 17:48:46,279 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:48:46,279 [I] Checking connection..
2026-09-29 17:48:46,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 17:48:46,526 [I] Checking connection passed.
2026-09-29 17:48:46,526 [I] --------- Verifying version history table ---------
2026-09-29 17:48:46,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:48:46,754 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:48:46,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:48:47,014 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:48:47,014 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 17:48:47,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 17:48:47.260 UTC [14677] WARNING: there is no transaction in progress
2026-09-29 17:48:47,265 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 17:48:47,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:48:47,501 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 17:48:47,501 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:48:47,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 17:48:47,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 17:48:47,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wbpt.c.gcp-upstream.internal','N') ;"
2026-09-29 17:48:48,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:48:48,482 [I] Table x_portal_user does not exist in database ranger
2026-09-29 17:48:48,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:48:48,730 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 17:48:48,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:48:48,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:48:49,230 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 17:48:49,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 17:48:50.229 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.246 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.263 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.279 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.295 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.313 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.381 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.388 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.402 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.413 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.424 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.438 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.448 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.455 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.461 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:50.467 UTC [14785] WARNING: there is no transaction in progress
2026-09-29 17:48:51,131 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 17:48:51,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 17:48:51,352 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 17:48:51,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:48:51,606 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 17:48:51,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:48:51,838 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 17:48:51,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:48:52,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:48:52,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wbpt.c.gcp-upstream.internal';"
2026-09-29 17:48:52,573 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 17:48:52,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 17:48:52,816 [I] DB_PATCHES have already been applied
I20260929 17:48:52.821338 3807 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-admin:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6qiWXY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 17:48:52.821401 3807 mini_ranger.cc:193] Using host: 127.3.183.212
I20260929 17:48:52.824615 3807 mini_ranger.cc:227] Ranger admin URL: http://127.3.183.212:40823
Sep 29, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 5:48:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 17:48:54.899297 3807 mini_ranger.cc:160] Time spent starting Ranger: real 9.455s user 0.003s sys 0.008s
Sep 29, 2026 5:49:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 5:49:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 5:49:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 5:49:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 5:49:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 17:49:16.827003 3807 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/rangerkms_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/rangerkms_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.183.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.183.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 17:49:16 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704156, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 17:49:16.909439 3807 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 17:49:16.920903 3807 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 17:49:17.815253 3807 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 17:49:17.815347 3807 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms
I20260929 17:49:17.815451 3807 mini_ranger_kms.cc:227] Using postgres at 127.3.183.212:55089
2026-09-29 17:49:18,520 [I] DB FLAVOR :POSTGRES
2026-09-29 17:49:18,520 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:49:18,520 [I] Checking connection
2026-09-29 17:49:18,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 17:49:18,880 [I] connection success
2026-09-29 17:49:18,880 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 17:49:18,880 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 17:49:18,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 17:49:19,123 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 17:49:19,123 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:49:19,123 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 17:49:19,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:55089/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 17:49:19,371 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 17:49:19.861488 3807 mini_ranger_kms.cc:310] Created kms service
I20260929 17:49:20.088769 3807 mini_ranger_kms.cc:326] Created kudu user
I20260929 17:49:20.132532 3807 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 17:49:20 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704156, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:49:20.574496 3807 mini_ranger_kms.cc:384] Added ranger policy
I20260929 17:49:20.574643 3807 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms:/tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 17:49:20.574662 3807 mini_ranger_kms.cc:240] Using host: 127.3.183.212
I20260929 17:49:20.578265 3807 mini_ranger_kms.cc:276] Ranger KMS PID: 14967
I20260929 17:49:20.578408 3807 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.183.212:56993
Sep 29, 2026 5:49:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:49:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:49:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 5:49:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 5:49:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 5:49:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 17:49:22.553462 3807 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.644s user 0.000s sys 0.674s
I20260929 17:49:22.553571 3807 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 17:49:22.553684 3807 mini_ranger_kms.cc:401] 127.3.183.212:56993/kms/v1/keys
Sep 29 17:49:23 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704163, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.183.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 17:49:23 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704163, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.183.212@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
Sep 29 17:49:23 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704163, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.183.212@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
Sep 29 17:49:26 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:49:26.393105 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:36553
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:36553
--ranger_config_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:56993/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:49:26.543431 15017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:49:26.543686 15017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:49:26.543725 15017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:49:26.547621 15017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:49:26.547672 15017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:49:26.547689 15017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:49:26.547704 15017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:49:26.547717 15017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:49:26.553215 15017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.183.212:56993/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:36553
--ranger_config_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:49:26.554141 15017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:49:26.555071 15017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:49:26.563685 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:49:26.563757 15017 server_base.cc:1034] running on GCE node
W20260929 17:49:26.563704 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:49:26.563743 15022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:49:26.564496 15017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:49:26.565513 15017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:49:26.566689 15017 hybrid_clock.cc:648] HybridClock initialized: now 1790704166566674 us; error 35 us; skew 500 ppm
Sep 29 17:49:26 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:49:26.574064 15017 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:49:26.574888 15017 webserver.cc:460] Webserver started at http://127.3.183.254:44277/ using document root <none> and password file <none>
I20260929 17:49:26.575374 15017 fs_manager.cc:359] Metadata directory not provided
I20260929 17:49:26.575428 15017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:49:26.575639 15017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:49:26 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:49:26.632129 15017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "0c769f36a33c4e8482e4b6dd22ebef15"
format_stamp: "Formatted at 2026-09-29 17:49:26 on dist-test-slave-wbpt"
server_key: "01ad77fcec7be871f1d2782beb74dec5"
server_key_iv: "2b52c41be37b518e0002178da41825c6"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:26.632782 15017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "0c769f36a33c4e8482e4b6dd22ebef15"
format_stamp: "Formatted at 2026-09-29 17:49:26 on dist-test-slave-wbpt"
server_key: "01ad77fcec7be871f1d2782beb74dec5"
server_key_iv: "2b52c41be37b518e0002178da41825c6"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:26.637276 15017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 17:49:26.650930 15032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:26.652362 15017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:49:26.652484 15017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "0c769f36a33c4e8482e4b6dd22ebef15"
format_stamp: "Formatted at 2026-09-29 17:49:26 on dist-test-slave-wbpt"
server_key: "01ad77fcec7be871f1d2782beb74dec5"
server_key_iv: "2b52c41be37b518e0002178da41825c6"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:26.652607 15017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:49:26.674538 15017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:49:26.677574 15017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:49:26.677796 15017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:49:26.692159 15017 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:36553
I20260929 17:49:26.692232 15084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:36553 every 8 connection(s)
I20260929 17:49:26.693316 15017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:49:26.696487 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:49:26.701130 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 15017
I20260929 17:49:26.701267 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:49:26.701562 3807 external_mini_cluster.cc:1428] Setting key 2b875dd6c651c25bdbf85201c15ef4ef
I20260929 17:49:26.703918 15085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15: Bootstrap starting.
I20260929 17:49:26.707329 15085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15: Neither blocks nor log segments found. Creating new log.
I20260929 17:49:26.708771 15085 log.cc:824] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15: Log is configured to *not* fsync() on all Append() calls
I20260929 17:49:26.710927 15085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15: No bootstrap required, opened a new log
Sep 29 17:49:26 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:49:26.716395 15085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36553 } }
I20260929 17:49:26.716655 15085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:49:26.716729 15085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c769f36a33c4e8482e4b6dd22ebef15, State: Initialized, Role: FOLLOWER
I20260929 17:49:26.717201 15085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36553 } }
I20260929 17:49:26.717335 15085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:49:26.717411 15085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:49:26.717527 15085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:49:26.718660 15085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36553 } }
I20260929 17:49:26.719013 15085 leader_election.cc:302] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c769f36a33c4e8482e4b6dd22ebef15; no voters:
I20260929 17:49:26.719344 15085 leader_election.cc:288] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:49:26.719762 15091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:49:26.720312 15091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [term 1 LEADER]: Becoming Leader. State: Replica: 0c769f36a33c4e8482e4b6dd22ebef15, State: Running, Role: LEADER
I20260929 17:49:26.720662 15091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36553 } }
I20260929 17:49:26.720745 15085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:49:26.721591 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:26.703223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.3.183.254:36553)
0929 17:49:26.703854 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:26.703899 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:26.703946 (+ 47us) server_negotiation.cc:407] Connection header received
0929 17:49:26.705190 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:26.705214 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:26.705688 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:26.706066 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:49:26.707189 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:26.708955 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:26.710312 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:26.711042 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:26.711186 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:26.714934 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:49:26.714947 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:49:26.714967 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:49:26.715005 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:49:26.717529 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:26.718123 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:26.718129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:26.718135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:26.718277 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:26.718619 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:26.718622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:26.718624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:26.719174 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:49:26.719355 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:26.720185 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 17:49:26.720489 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":167,"threads_started":1}
I20260929 17:49:26.722671 15091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c769f36a33c4e8482e4b6dd22ebef15. Latest consensus state: current_term: 1 leader_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36553 } } }
I20260929 17:49:26.722749 15090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c769f36a33c4e8482e4b6dd22ebef15" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36553 } } }
I20260929 17:49:26.722782 15091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [sys.catalog]: This master's current role is: LEADER
I20260929 17:49:26.722949 15090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15 [sys.catalog]: This master's current role is: LEADER
I20260929 17:49:26.723448 15099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:49:26.727193 15099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:49:26.727746 15085 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 17:49:26.741283 15107 mvcc.cc:205] Tried to move back new op lower bound from 7334724266932776960 to 7334724266888937472. Current Snapshot: MvccSnapshot[applied={T|T < 7334724266932776960}]
I20260929 17:49:26.743495 15099 catalog_manager.cc:1347] Generated new cluster ID: c0eb40882090410aa8a79e448244c745
I20260929 17:49:26.743567 15099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:49:26.800191 15099 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:49:26.801147 15099 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:49:26.810367 15099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15: Generated new TSK 0
I20260929 17:49:26.811081 15099 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:49:27.715659 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:56993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36553
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:49:27.893168 15130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:49:27.893415 15130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:49:27.893477 15130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:49:27.898121 15130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:49:27.898181 15130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:49:27.898262 15130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:49:27.904073 15130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.183.212:56993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:49:27.905128 15130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:49:27.906136 15130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:49:27.916042 15140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:49:27.916096 15137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:49:27.916416 15138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:49:27.916404 15130 server_base.cc:1034] running on GCE node
I20260929 17:49:27.916888 15130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:49:27.917683 15130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:49:27.918884 15130 hybrid_clock.cc:648] HybridClock initialized: now 1790704167918858 us; error 38 us; skew 500 ppm
Sep 29 17:49:27 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:49:27.923015 15130 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:49:27.923908 15130 webserver.cc:460] Webserver started at http://127.3.183.193:35785/ using document root <none> and password file <none>
I20260929 17:49:27.924437 15130 fs_manager.cc:359] Metadata directory not provided
I20260929 17:49:27.924502 15130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:49:27.924734 15130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:49:27 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:49:27.945696 15130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "905686494faf43699bddfa7d03a4b641"
format_stamp: "Formatted at 2026-09-29 17:49:27 on dist-test-slave-wbpt"
server_key: "31cd95ca600691568749637895330499"
server_key_iv: "58402a877b83d51e369a9a614faffb03"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:27.946583 15130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "905686494faf43699bddfa7d03a4b641"
format_stamp: "Formatted at 2026-09-29 17:49:27 on dist-test-slave-wbpt"
server_key: "31cd95ca600691568749637895330499"
server_key_iv: "58402a877b83d51e369a9a614faffb03"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:27.952185 15130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 17:49:27.970296 15147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:27.972154 15130 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:49:27.972313 15130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "905686494faf43699bddfa7d03a4b641"
format_stamp: "Formatted at 2026-09-29 17:49:27 on dist-test-slave-wbpt"
server_key: "31cd95ca600691568749637895330499"
server_key_iv: "58402a877b83d51e369a9a614faffb03"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:27.972489 15130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:49:28.008790 15130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:49:28.012879 15130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:49:28.013219 15130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:49:28.014101 15130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:49:28.015272 15130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:49:28.015383 15130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.015499 15130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:49:28.015573 15130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.038306 15130 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:42287
I20260929 17:49:28.039685 15130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:49:28.041009 15262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:42287 every 8 connection(s)
I20260929 17:49:28.044191 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 15130
I20260929 17:49:28.044320 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:49:28.044654 3807 external_mini_cluster.cc:1428] Setting key 1be7bfe04a2cbb7cad634952bf192eb3
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:49:28.072207 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.044012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:48457 (local address 127.3.183.254:36553)
0929 17:49:28.053383 (+ 9371us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:28.053408 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:28.055624 (+ 2216us) server_negotiation.cc:407] Connection header received
0929 17:49:28.055697 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:28.055700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:28.055765 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:28.055906 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:49:28.057900 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.058760 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.060667 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.061033 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.061134 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.064881 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:49:28.064885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:49:28.064888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:49:28.064916 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:49:28.066790 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.067452 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.067456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.067457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.067554 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.068151 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.068154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.068156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.068381 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:49:28.068485 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:28.071911 (+ 3426us) server_negotiation.cc:299] Negotiation successful
0929 17:49:28.072037 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9261,"thread_start_us":93,"threads_started":1}
I20260929 17:49:28.073881 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.043705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36553 (local address 127.3.183.193:48457)
0929 17:49:28.053818 (+ 10113us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:28.053905 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:28.055393 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:28.056146 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:28.056157 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:28.056757 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:49:28.057704 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.057727 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.058918 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.058922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.060523 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.060534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.061338 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.061341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.061600 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.062472 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:49:28.062494 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:49:28.064676 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:49:28.066924 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:49:28.066931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:49:28.066950 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:49:28.067326 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:49:28.067849 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:49:28.067853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:49:28.067855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:49:28.068039 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:49:28.068672 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:49:28.068679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:49:28.069041 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 17:49:28.071447 (+ 2406us) client_negotiation.cc:240] Negotiation successful
0929 17:49:28.072674 (+ 1227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9796,"thread_start_us":232,"threads_started":1}
I20260929 17:49:28.075649 15263 heartbeater.cc:347] Connected to a master server at 127.3.183.254:36553
I20260929 17:49:28.075927 15263 heartbeater.cc:464] Registering TS with master...
I20260929 17:49:28.076507 15263 heartbeater.cc:511] Master 127.3.183.254:36553 requested a full tablet report, sending...
I20260929 17:49:28.078317 15049 ts_manager.cc:194] Registered new tserver with Master: 905686494faf43699bddfa7d03a4b641 (127.3.183.193:42287)
I20260929 17:49:28.080040 15049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:48457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:49:28.104002 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:56993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36553
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 17:49:28.331689 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:49:28.332137 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:49:28.332266 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:49:28.338377 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:49:28.338466 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:49:28.338708 15269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:49:28.347360 15269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.183.212:56993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:49:28.348923 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:49:28.350371 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:49:28.363442 15275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:49:28.363431 15278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:49:28.363688 15276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:49:28.364238 15269 server_base.cc:1034] running on GCE node
I20260929 17:49:28.364674 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:49:28.365337 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:49:28.366585 15269 hybrid_clock.cc:648] HybridClock initialized: now 1790704168366558 us; error 38 us; skew 500 ppm
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:49:28.369961 15269 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:49:28.370867 15269 webserver.cc:460] Webserver started at http://127.3.183.194:37297/ using document root <none> and password file <none>
I20260929 17:49:28.371443 15269 fs_manager.cc:359] Metadata directory not provided
I20260929 17:49:28.371510 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:49:28.371739 15269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:49:28.388172 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "3016e7e144b441318dfa80f5f46c11ca"
format_stamp: "Formatted at 2026-09-29 17:49:28 on dist-test-slave-wbpt"
server_key: "d4c465f7ed88322f66c2307bc4c9a74e"
server_key_iv: "3337b49f3f310fb13c76e0b32ceae751"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:28.389174 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "3016e7e144b441318dfa80f5f46c11ca"
format_stamp: "Formatted at 2026-09-29 17:49:28 on dist-test-slave-wbpt"
server_key: "d4c465f7ed88322f66c2307bc4c9a74e"
server_key_iv: "3337b49f3f310fb13c76e0b32ceae751"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:28.401239 15269 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.000s sys 0.004s
I20260929 17:49:28.436275 15285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.438364 15269 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.001s sys 0.001s
I20260929 17:49:28.438621 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "3016e7e144b441318dfa80f5f46c11ca"
format_stamp: "Formatted at 2026-09-29 17:49:28 on dist-test-slave-wbpt"
server_key: "d4c465f7ed88322f66c2307bc4c9a74e"
server_key_iv: "3337b49f3f310fb13c76e0b32ceae751"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:28.438879 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:49:28.500140 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:49:28.504077 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:49:28.504359 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:49:28.505302 15269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:49:28.506523 15269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:49:28.506587 15269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.506676 15269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:49:28.506711 15269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.531055 15269 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:45103
I20260929 17:49:28.531073 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:45103 every 8 connection(s)
I20260929 17:49:28.532500 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:49:28.535290 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 15269
I20260929 17:49:28.535414 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:49:28.535676 3807 external_mini_cluster.cc:1428] Setting key feee4fddc7a218054ce81a51eee38d64
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:49:28.553557 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.535100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36553 (local address 127.3.183.194:33809)
0929 17:49:28.536246 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:28.536301 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:28.537327 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:28.537890 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:28.537899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:28.538224 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:49:28.538894 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.538916 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.542765 (+ 3849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.542769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.544239 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.544251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.544973 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.544978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.545183 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.545855 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:49:28.545876 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:49:28.547669 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:49:28.550435 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:49:28.550441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:49:28.550457 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:49:28.550770 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:49:28.551045 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:49:28.551047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:49:28.551049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:49:28.551203 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:49:28.551665 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:49:28.551671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:49:28.551997 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 17:49:28.552192 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 17:49:28.552513 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":213,"threads_started":1}
I20260929 17:49:28.554237 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.534554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:33809 (local address 127.3.183.254:36553)
0929 17:49:28.534701 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:28.534705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:28.536418 (+ 1713us) server_negotiation.cc:407] Connection header received
0929 17:49:28.537555 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:28.537559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:28.537614 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:28.537709 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:49:28.540330 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.541236 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.544361 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.544799 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.544918 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.548779 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:49:28.548783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:49:28.548785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:49:28.548808 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:49:28.550310 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.550875 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.550879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.550880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.550960 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.551295 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.551299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.551300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.551512 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:49:28.551615 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:28.553953 (+ 2338us) server_negotiation.cc:299] Negotiation successful
0929 17:49:28.554085 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:49:28.556006 15399 heartbeater.cc:347] Connected to a master server at 127.3.183.254:36553
I20260929 17:49:28.556267 15399 heartbeater.cc:464] Registering TS with master...
I20260929 17:49:28.556905 15399 heartbeater.cc:511] Master 127.3.183.254:36553 requested a full tablet report, sending...
I20260929 17:49:28.558368 15049 ts_manager.cc:194] Registered new tserver with Master: 3016e7e144b441318dfa80f5f46c11ca (127.3.183.194:45103)
I20260929 17:49:28.559192 15049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:33809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:49:28.571762 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:56993/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36553
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:49:28.784947 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:49:28.785441 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:49:28.785552 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:49:28.791348 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:49:28.791465 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:49:28.791575 15404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:49:28.800060 15404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.183.212:56993/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:36553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:49:28.801754 15404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:49:28.803407 15404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
W20260929 17:49:28.818699 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:49:28.818799 15404 server_base.cc:1034] running on GCE node
W20260929 17:49:28.819139 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:49:28.819360 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:49:28.819976 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:49:28.820725 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:49:28.821882 15404 hybrid_clock.cc:648] HybridClock initialized: now 1790704168821799 us; error 96 us; skew 500 ppm
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:49:28.826737 15404 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:49:28.827968 15404 webserver.cc:460] Webserver started at http://127.3.183.195:41999/ using document root <none> and password file <none>
I20260929 17:49:28.828738 15404 fs_manager.cc:359] Metadata directory not provided
I20260929 17:49:28.828881 15404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:49:28.829154 15404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:49:28.853197 15404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "d9b5eeeb6a854bcaae34e77fad108eaa"
format_stamp: "Formatted at 2026-09-29 17:49:28 on dist-test-slave-wbpt"
server_key: "e557ebd447800134537df9a6c4cee9de"
server_key_iv: "cad07e5e723ff0db5ec0d56d3a172905"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:28.854034 15404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "d9b5eeeb6a854bcaae34e77fad108eaa"
format_stamp: "Formatted at 2026-09-29 17:49:28 on dist-test-slave-wbpt"
server_key: "e557ebd447800134537df9a6c4cee9de"
server_key_iv: "cad07e5e723ff0db5ec0d56d3a172905"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:28.860548 15404 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 17:49:28.878485 15419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.880393 15404 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:49:28.880524 15404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "d9b5eeeb6a854bcaae34e77fad108eaa"
format_stamp: "Formatted at 2026-09-29 17:49:28 on dist-test-slave-wbpt"
server_key: "e557ebd447800134537df9a6c4cee9de"
server_key_iv: "cad07e5e723ff0db5ec0d56d3a172905"
server_key_version: "kuduclusterkey@0"
I20260929 17:49:28.880662 15404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:49:28.897667 15404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:49:28.900838 15404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:49:28.901021 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:49:28.901657 15404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:49:28.902603 15404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:49:28.902652 15404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.902702 15404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:49:28.902726 15404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:49:28.921725 15404 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:43527
I20260929 17:49:28.922919 15404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:49:28.923337 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 15404
I20260929 17:49:28.923491 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:49:28.923884 3807 external_mini_cluster.cc:1428] Setting key cf7dc1fe6daa2b1e7957d38ceee4c3f4
I20260929 17:49:28.934465 15532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:43527 every 8 connection(s)
Sep 29 17:49:28 dist-test-slave-wbpt krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704168, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:49:28.951031 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.935935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:60323 (local address 127.3.183.254:36553)
0929 17:49:28.936080 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:28.936085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:28.938044 (+ 1959us) server_negotiation.cc:407] Connection header received
0929 17:49:28.939316 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:28.939319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:28.939388 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:28.939466 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:49:28.940895 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.941435 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.942695 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.942934 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.943029 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.946331 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:49:28.946336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:49:28.946339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:49:28.946366 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:49:28.947952 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.948894 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.948898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.948900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.949118 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.949546 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.949549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.949551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.949975 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:49:28.950053 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:28.950749 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 17:49:28.950875 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:49:28.952169 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.936406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36553 (local address 127.3.183.195:60323)
0929 17:49:28.937884 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:28.937948 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:28.939104 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:28.939667 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:28.939674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:28.940132 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:49:28.940748 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.940765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.941558 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.941561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.942572 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.942582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.943077 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.943082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.943297 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.943996 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:49:28.944017 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:49:28.946149 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:49:28.948085 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:49:28.948092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:49:28.948108 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:49:28.948586 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:49:28.949222 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:49:28.949225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:49:28.949227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:49:28.949444 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:49:28.950116 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:49:28.950122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:49:28.950441 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 17:49:28.950634 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 17:49:28.950958 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"thread_start_us":260,"threads_started":1}
I20260929 17:49:28.953411 15533 heartbeater.cc:347] Connected to a master server at 127.3.183.254:36553
I20260929 17:49:28.953722 15533 heartbeater.cc:464] Registering TS with master...
I20260929 17:49:28.954298 15533 heartbeater.cc:511] Master 127.3.183.254:36553 requested a full tablet report, sending...
I20260929 17:49:28.955466 15049 ts_manager.cc:194] Registered new tserver with Master: d9b5eeeb6a854bcaae34e77fad108eaa (127.3.183.195:43527)
I20260929 17:49:28.956218 15049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:60323
I20260929 17:49:28.961699 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:49:28.971850 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.963952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.3.183.254:36553)
0929 17:49:28.964073 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:28.964077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:28.964346 (+ 269us) server_negotiation.cc:407] Connection header received
0929 17:49:28.964475 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:28.964477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:28.964515 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:28.964581 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:49:28.965407 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.965950 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.967148 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.967357 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.967429 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.969055 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:49:28.969058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:49:28.969060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:49:28.969085 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:49:28.970358 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.970784 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.970786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.970787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.970854 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:49:28.971104 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:49:28.971106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:49:28.971107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:49:28.971254 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:49:28.971343 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:28.971596 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 17:49:28.971700 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:49:28.974598 15049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:49:28.978011 15049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:49:28.997274 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.987981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:43527 (local address 127.0.0.1:60858)
0929 17:49:28.989735 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:28.989783 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:28.989901 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:28.992528 (+ 2627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:28.992531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:28.992548 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:49:28.992663 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.992669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.994195 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.994199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.995444 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.995454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.996681 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.996685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.996807 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.996861 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 17:49:28.996953 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 17:49:28.997067 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"mutex_wait_us":26,"thread_start_us":125,"threads_started":1}
I20260929 17:49:28.997438 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.987971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42287 (local address 127.0.0.1:39850)
0929 17:49:28.990274 (+ 2303us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:28.990286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:28.990359 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:28.991084 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:28.991092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:28.991120 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:49:28.991355 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.991363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.992907 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.992912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.995570 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:28.995574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.996830 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.996834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:28.996977 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.997005 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 17:49:28.997062 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 17:49:28.997148 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2215,"mutex_wait_us":1069,"thread_start_us":416,"threads_started":1}
I20260929 17:49:28.998214 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.990255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.3.183.193:42287)
0929 17:49:28.990575 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:28.990583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:28.990605 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:49:28.990676 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:28.990682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:28.990878 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:28.991034 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:49:28.991516 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.992766 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.995666 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.996701 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.997763 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.997914 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:28.997964 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 17:49:28.998059 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":119,"threads_started":1}
I20260929 17:49:28.998608 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.988083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.3.183.195:43527)
0929 17:49:28.991332 (+ 3249us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:28.991343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:28.991363 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:49:28.992176 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:28.992183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:28.992378 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:28.992795 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:49:28.992921 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.994065 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.995705 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:28.996546 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:28.998009 (+ 1463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:28.998152 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:28.998198 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 17:49:28.998469 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3150,"thread_start_us":141,"threads_started":1}
I20260929 17:49:29.001724 15467 tablet_service.cc:1467] Processing CreateTablet for tablet a382e187f13f4594a5a1a5f61c21a68b (DEFAULT_TABLE table=test-table [id=d45ff695a8ff4ae89470629798076f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:49:29.001748 15196 tablet_service.cc:1467] Processing CreateTablet for tablet a382e187f13f4594a5a1a5f61c21a68b (DEFAULT_TABLE table=test-table [id=d45ff695a8ff4ae89470629798076f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:49:29.002853 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a382e187f13f4594a5a1a5f61c21a68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:49:29.002853 15196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a382e187f13f4594a5a1a5f61c21a68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:49:29.012895 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.989060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45103 (local address 127.0.0.1:50690)
0929 17:49:28.997423 (+ 8363us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:28.997436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:28.997498 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:29.007049 (+ 9551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:29.007053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:29.007073 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:49:29.007175 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:29.007181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.009070 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.009074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:29.010252 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:29.010265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.012507 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.012511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:29.012604 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:29.012632 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 17:49:29.012678 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 17:49:29.012754 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8312,"thread_start_us":83,"threads_started":1}
I20260929 17:49:29.013828 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:28.989231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.3.183.194:45103)
0929 17:49:29.006455 (+ 17224us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:29.006508 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:29.006537 (+ 29us) server_negotiation.cc:407] Connection header received
0929 17:49:29.006618 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:29.006623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:29.006835 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:29.007336 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:49:29.007545 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.008875 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.011343 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.012361 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.013356 (+ 995us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:29.013525 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:29.013574 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 17:49:29.013665 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17087,"thread_start_us":86,"threads_started":1}
I20260929 17:49:29.023639 15333 tablet_service.cc:1467] Processing CreateTablet for tablet a382e187f13f4594a5a1a5f61c21a68b (DEFAULT_TABLE table=test-table [id=d45ff695a8ff4ae89470629798076f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:49:29.025143 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a382e187f13f4594a5a1a5f61c21a68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:49:29.027119 15551 tablet_bootstrap.cc:492] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Bootstrap starting.
I20260929 17:49:29.032065 15551 tablet_bootstrap.cc:654] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Neither blocks nor log segments found. Creating new log.
I20260929 17:49:29.032383 15552 tablet_bootstrap.cc:492] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641: Bootstrap starting.
I20260929 17:49:29.033484 15551 log.cc:824] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Log is configured to *not* fsync() on all Append() calls
I20260929 17:49:29.034552 15554 tablet_bootstrap.cc:492] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca: Bootstrap starting.
I20260929 17:49:29.035585 15552 tablet_bootstrap.cc:654] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641: Neither blocks nor log segments found. Creating new log.
I20260929 17:49:29.036914 15552 log.cc:824] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641: Log is configured to *not* fsync() on all Append() calls
I20260929 17:49:29.037603 15554 tablet_bootstrap.cc:654] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca: Neither blocks nor log segments found. Creating new log.
I20260929 17:49:29.039253 15551 tablet_bootstrap.cc:492] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: No bootstrap required, opened a new log
I20260929 17:49:29.039287 15552 tablet_bootstrap.cc:492] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641: No bootstrap required, opened a new log
I20260929 17:49:29.039458 15552 ts_tablet_manager.cc:1397] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 17:49:29.039467 15551 ts_tablet_manager.cc:1397] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260929 17:49:29.039546 15554 log.cc:824] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca: Log is configured to *not* fsync() on all Append() calls
I20260929 17:49:29.043293 15554 tablet_bootstrap.cc:492] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca: No bootstrap required, opened a new log
I20260929 17:49:29.043489 15554 ts_tablet_manager.cc:1397] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 17:49:29.049711 15554 raft_consensus.cc:348] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.050215 15554 raft_consensus.cc:374] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:49:29.050412 15554 raft_consensus.cc:729] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3016e7e144b441318dfa80f5f46c11ca, State: Initialized, Role: FOLLOWER
I20260929 17:49:29.051136 15554 consensus_queue.cc:260] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.052593 15554 ts_tablet_manager.cc:1428] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca: Time spent starting tablet: real 0.009s user 0.001s sys 0.008s
I20260929 17:49:29.053054 15399 heartbeater.cc:503] Master 127.3.183.254:36553 was elected leader, sending a full tablet report...
I20260929 17:49:29.053488 15551 raft_consensus.cc:348] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.053757 15551 raft_consensus.cc:374] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:49:29.053803 15551 raft_consensus.cc:729] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b5eeeb6a854bcaae34e77fad108eaa, State: Initialized, Role: FOLLOWER
I20260929 17:49:29.054329 15551 consensus_queue.cc:260] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.057343 15552 raft_consensus.cc:348] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.059854 15552 raft_consensus.cc:374] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:49:29.059998 15552 raft_consensus.cc:729] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905686494faf43699bddfa7d03a4b641, State: Initialized, Role: FOLLOWER
I20260929 17:49:29.060707 15552 consensus_queue.cc:260] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.061975 15552 ts_tablet_manager.cc:1428] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641: Time spent starting tablet: real 0.022s user 0.010s sys 0.000s
I20260929 17:49:29.062914 15551 ts_tablet_manager.cc:1428] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20260929 17:49:29.063261 15533 heartbeater.cc:503] Master 127.3.183.254:36553 was elected leader, sending a full tablet report...
I20260929 17:49:29.065927 15263 heartbeater.cc:503] Master 127.3.183.254:36553 was elected leader, sending a full tablet report...
I20260929 17:49:29.070668 15560 raft_consensus.cc:482] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:49:29.071120 15560 raft_consensus.cc:504] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.075551 15560 leader_election.cc:288] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9b5eeeb6a854bcaae34e77fad108eaa (127.3.183.195:43527), 3016e7e144b441318dfa80f5f46c11ca (127.3.183.194:45103)
I20260929 17:49:29.080255 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:29.074585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:51813 (local address 127.3.183.195:43527)
0929 17:49:29.074708 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:29.074712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:29.075398 (+ 686us) server_negotiation.cc:407] Connection header received
0929 17:49:29.075599 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:29.075602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:29.075652 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:29.075700 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:49:29.076115 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.076838 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.078727 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.079281 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.079364 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:29.079526 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:29.080048 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 17:49:29.080126 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:49:29.080482 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:29.075400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:57793 (local address 127.3.183.194:45103)
0929 17:49:29.075518 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:29.075523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:29.075997 (+ 474us) server_negotiation.cc:407] Connection header received
0929 17:49:29.076163 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:29.076166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:29.076210 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:29.076253 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:49:29.076656 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.077369 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.079137 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.079818 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.079903 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:29.079949 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:29.080289 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 17:49:29.080372 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:49:29.080803 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:29.074482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:43527 (local address 127.3.183.193:51813)
0929 17:49:29.075294 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:29.075308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:29.075482 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:29.075864 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:29.075867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:29.075886 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:49:29.075991 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:29.075997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.077057 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.077060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:29.078578 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:29.078588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.079823 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.079827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:29.079911 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:29.079933 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:49:29.080438 (+ 505us) client_negotiation.cc:240] Negotiation successful
0929 17:49:29.080507 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":150,"threads_started":1}
I20260929 17:49:29.081434 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:29.075001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45103 (local address 127.3.183.193:57793)
0929 17:49:29.075891 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:29.075912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 17:49:29.076055 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:49:29.076373 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:49:29.076375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:49:29.076390 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:49:29.076537 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:29.076544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.077497 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.077500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:29.079016 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:49:29.079025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.080076 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.080080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:49:29.080157 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:29.080178 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:49:29.081165 (+ 987us) client_negotiation.cc:240] Negotiation successful
0929 17:49:29.081239 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":135,"threads_started":1}
I20260929 17:49:29.082209 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382e187f13f4594a5a1a5f61c21a68b" candidate_uuid: "905686494faf43699bddfa7d03a4b641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3016e7e144b441318dfa80f5f46c11ca" is_pre_election: true
I20260929 17:49:29.082540 15353 raft_consensus.cc:2454] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905686494faf43699bddfa7d03a4b641 in term 0.
I20260929 17:49:29.083244 15149 leader_election.cc:302] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3016e7e144b441318dfa80f5f46c11ca, 905686494faf43699bddfa7d03a4b641; no voters:
I20260929 17:49:29.083639 15560 raft_consensus.cc:2781] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:49:29.083731 15560 raft_consensus.cc:482] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:49:29.083796 15560 raft_consensus.cc:3037] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:49:29.085742 15560 raft_consensus.cc:504] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.086408 15560 leader_election.cc:288] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [CANDIDATE]: Term 1 election: Requested vote from peers d9b5eeeb6a854bcaae34e77fad108eaa (127.3.183.195:43527), 3016e7e144b441318dfa80f5f46c11ca (127.3.183.194:45103)
I20260929 17:49:29.086836 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382e187f13f4594a5a1a5f61c21a68b" candidate_uuid: "905686494faf43699bddfa7d03a4b641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3016e7e144b441318dfa80f5f46c11ca"
I20260929 17:49:29.086812 15475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382e187f13f4594a5a1a5f61c21a68b" candidate_uuid: "905686494faf43699bddfa7d03a4b641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" is_pre_election: true
I20260929 17:49:29.086953 15353 raft_consensus.cc:3037] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:49:29.087095 15475 raft_consensus.cc:2454] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905686494faf43699bddfa7d03a4b641 in term 0.
I20260929 17:49:29.087519 15476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382e187f13f4594a5a1a5f61c21a68b" candidate_uuid: "905686494faf43699bddfa7d03a4b641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa"
I20260929 17:49:29.087633 15476 raft_consensus.cc:3037] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:49:29.088625 15353 raft_consensus.cc:2454] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905686494faf43699bddfa7d03a4b641 in term 1.
I20260929 17:49:29.089426 15476 raft_consensus.cc:2454] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905686494faf43699bddfa7d03a4b641 in term 1.
I20260929 17:49:29.090041 15148 leader_election.cc:302] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 905686494faf43699bddfa7d03a4b641, d9b5eeeb6a854bcaae34e77fad108eaa; no voters:
I20260929 17:49:29.090831 15560 raft_consensus.cc:2781] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:49:29.091223 15560 raft_consensus.cc:686] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [term 1 LEADER]: Becoming Leader. State: Replica: 905686494faf43699bddfa7d03a4b641, State: Running, Role: LEADER
I20260929 17:49:29.091621 15560 consensus_queue.cc:237] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } }
I20260929 17:49:29.095443 15048 catalog_manager.cc:5495] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 reported cstate change: term changed from 0 to 1, leader changed from <none> to 905686494faf43699bddfa7d03a4b641 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "905686494faf43699bddfa7d03a4b641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905686494faf43699bddfa7d03a4b641" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 } health_report { overall_health: UNKNOWN } } }
I20260929 17:49:29.124680 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:29.119906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.3.183.193:42287)
0929 17:49:29.120041 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:49:29.120046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:49:29.120060 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:49:29.120147 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:49:29.120150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:49:29.120214 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:49:29.120311 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:49:29.120682 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.121479 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.122768 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:49:29.123162 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:49:29.123254 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:49:29.123664 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:49:29.124176 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:49:29.124283 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:49:29.124374 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 17:49:29.124432 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 17:49:29.130857 15476 raft_consensus.cc:1261] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 1 FOLLOWER]: Refusing update from remote peer 905686494faf43699bddfa7d03a4b641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:49:29.130903 15353 raft_consensus.cc:1261] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 1 FOLLOWER]: Refusing update from remote peer 905686494faf43699bddfa7d03a4b641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:49:29.131698 15559 consensus_queue.cc:1035] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b5eeeb6a854bcaae34e77fad108eaa" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:49:29.132117 15559 consensus_queue.cc:1035] T a382e187f13f4594a5a1a5f61c21a68b P 905686494faf43699bddfa7d03a4b641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3016e7e144b441318dfa80f5f46c11ca" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:49:29.145833 15566 mvcc.cc:205] Tried to move back new op lower bound from 7334724276752637952 to 7334724276603346944. Current Snapshot: MvccSnapshot[applied={T|T < 7334724276752637952}]
I20260929 17:49:29.149832 15567 mvcc.cc:205] Tried to move back new op lower bound from 7334724276752637952 to 7334724276603346944. Current Snapshot: MvccSnapshot[applied={T|T < 7334724276752637952}]
I20260929 17:49:29.160849 15049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:49:29.161031 15049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:49:29.164137 15049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c769f36a33c4e8482e4b6dd22ebef15: Sending DeleteTablet for 3 replicas of tablet a382e187f13f4594a5a1a5f61c21a68b
I20260929 17:49:29.165310 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet a382e187f13f4594a5a1a5f61c21a68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:49:29 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:60858
I20260929 17:49:29.166735 15196 tablet_service.cc:1514] Processing DeleteTablet for tablet a382e187f13f4594a5a1a5f61c21a68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:49:29 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:39850
I20260929 17:49:29.167461 15333 tablet_service.cc:1514] Processing DeleteTablet for tablet a382e187f13f4594a5a1a5f61c21a68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:49:29 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:50690
I20260929 17:49:29.168960 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 15130
I20260929 17:49:29.171285 15577 tablet_replica.cc:330] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca: stopping tablet replica
I20260929 17:49:29.171677 15577 raft_consensus.cc:2229] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:49:29.172075 15575 tablet_replica.cc:330] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: stopping tablet replica
I20260929 17:49:29.172477 15575 raft_consensus.cc:2229] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:49:29.172971 15577 raft_consensus.cc:2258] T a382e187f13f4594a5a1a5f61c21a68b P 3016e7e144b441318dfa80f5f46c11ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:49:29.173337 15575 raft_consensus.cc:2258] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:49:29.175437 15575 ts_tablet_manager.cc:1905] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:49:29.178439 15575 ts_tablet_manager.cc:1918] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 17:49:29.178535 15034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:42287 (error 108)
I20260929 17:49:29.179510 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 15269
I20260929 17:49:29.180303 15575 log.cc:1196] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/a382e187f13f4594a5a1a5f61c21a68b
I20260929 17:49:29.180665 15575 ts_tablet_manager.cc:1939] T a382e187f13f4594a5a1a5f61c21a68b P d9b5eeeb6a854bcaae34e77fad108eaa: Deleting consensus metadata
I20260929 17:49:29.184122 15033 catalog_manager.cc:4841] TS d9b5eeeb6a854bcaae34e77fad108eaa (127.3.183.195:43527): tablet a382e187f13f4594a5a1a5f61c21a68b (table test-table [id=d45ff695a8ff4ae89470629798076f82]) successfully deleted
I20260929 17:49:29.184958 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:29.184593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42287 (local address 127.0.0.1:39858)
0929 17:49:29.184784 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:29.184865 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260929 17:49:29.185184 15034 catalog_manager.cc:4584] TS 905686494faf43699bddfa7d03a4b641 (127.3.183.193:42287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a382e187f13f4594a5a1a5f61c21a68b: Network error: Client connection negotiation failed: client connection to 127.3.183.193:42287: connect: Connection refused (error 111)
I20260929 17:49:29.187436 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 15404
I20260929 17:49:29.189704 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:49:29.187607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45103 (local address 127.0.0.1:50704)
0929 17:49:29.189560 (+ 1953us) negotiation.cc:107] Waiting for socket to connect
0929 17:49:29.189622 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1851}
I20260929 17:49:29.194886 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 15017
I20260929 17:49:29.201025 3807 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 17:49:29.536026 3807 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 17:49:29.536154 3807 mini_ranger.cc:66] Stopping Ranger...
I20260929 17:49:29.871289 3807 mini_ranger.cc:68] Stopped Ranger
2026-09-29 17:49:29.871 UTC [14602] LOG: received smart shutdown request
2026-09-29 17:49:29.875 UTC [14602] LOG: background worker "logical replication launcher" (PID 14611) exited with exit code 1
2026-09-29 17:49:29.876 UTC [14606] LOG: shutting down
2026-09-29 17:49:30.053 UTC [14602] LOG: database system is shut down
2026-09-29T17:49:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49376 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:49:30 dist-test-slave-wbpt krb5kdc[15587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:49:30 dist-test-slave-wbpt krb5kdc[15587](info): set up 2 sockets
Sep 29 17:49:30 dist-test-slave-wbpt krb5kdc[15587](info): commencing operation
krb5kdc: starting...
W20260929 17:49:31.115612 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:49:31 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:49:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/rangeradmin_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/rangeradmin_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/rangerlookup_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/rangerlookup_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
I20260929 17:49:31.274386 3807 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6qiWXY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/postgres -l logfile start
2026-09-29 17:49:33.740 UTC [15617] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 17:49:33.740 UTC [15617] LOG: listening on IPv4 address "127.3.183.212", port 46993
2026-09-29 17:49:33.744 UTC [15617] LOG: listening on Unix socket "/tmp/.s.PGSQL.46993"
2026-09-29 17:49:33.763 UTC [15620] LOG: database system was shut down at 2026-09-29 17:49:32 UTC
2026-09-29 17:49:33.766 UTC [15617] LOG: database system is ready to accept connections
I20260929 17:49:34.643685 3807 mini_postgres.cc:96] Postgres bound to 46993
2026-09-29 17:49:34.649 UTC [15628] FATAL: database "slave" does not exist
127.3.183.212:46993 - accepting connections
I20260929 17:49:34.650144 3807 mini_ranger.cc:161] Starting Ranger...
I20260929 17:49:34.664852 3807 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 17:49:35.152724 3807 mini_ranger.cc:87] Created ranger Postgres database
I20260929 17:49:35.152804 3807 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-admin
2026-09-29 17:49:35,425 [I] DB FLAVOR :POSTGRES
2026-09-29 17:49:35,426 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:49:35,426 [I] Checking connection..
2026-09-29 17:49:35,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 17:49:35,682 [I] Checking connection passed.
2026-09-29 17:49:35,683 [I] --------- Verifying version history table ---------
2026-09-29 17:49:35,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:49:35,920 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:49:35,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:49:36,165 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:49:36,165 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 17:49:36,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 17:49:36.400 UTC [15692] WARNING: there is no transaction in progress
2026-09-29 17:49:36,406 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 17:49:36,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:49:36,662 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 17:49:36,662 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:49:36,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 17:49:36,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 17:49:37,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wbpt.c.gcp-upstream.internal','N') ;"
2026-09-29 17:49:37,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:49:37,630 [I] Table x_portal_user does not exist in database ranger
2026-09-29 17:49:37,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:49:37,884 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 17:49:37,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:49:38,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:49:38,378 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 17:49:38,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 17:49:39.401 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.419 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.436 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.453 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.471 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.489 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.548 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.556 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.570 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.581 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.592 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.607 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.617 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.623 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.630 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:39.637 UTC [15800] WARNING: there is no transaction in progress
2026-09-29 17:49:40,334 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 17:49:40,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 17:49:40,567 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 17:49:40,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:49:40,809 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 17:49:40,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:49:41,045 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 17:49:41,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:49:41,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:49:41,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wbpt.c.gcp-upstream.internal';"
2026-09-29 17:49:41,777 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 17:49:41,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 17:49:42,016 [I] DB_PATCHES have already been applied
I20260929 17:49:42.021996 3807 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-admin:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6qiWXY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 17:49:42.022071 3807 mini_ranger.cc:193] Using host: 127.3.183.212
I20260929 17:49:42.025688 3807 mini_ranger.cc:227] Ranger admin URL: http://127.3.183.212:42421
Sep 29, 2026 5:49:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:49:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:49:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 5:49:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 5:49:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 5:49:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 5:49:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 17:49:43.020920 3807 mini_ranger.cc:160] Time spent starting Ranger: real 8.371s user 0.004s sys 0.006s
Sep 29, 2026 5:50:03 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 5:50:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 5:50:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 5:50:03 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 5:50:04 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 17:50:05.008486 3807 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.183.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.183.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/rangerkms_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.183.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/rangerkms_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.183.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.183.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.183.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/HTTP_127.3.183.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 17:50:05 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704205, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 17:50:05.092901 3807 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 17:50:05.102182 3807 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 17:50:05.918923 3807 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 17:50:05.919005 3807 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms
I20260929 17:50:05.919021 3807 mini_ranger_kms.cc:227] Using postgres at 127.3.183.212:46993
2026-09-29 17:50:06,077 [I] DB FLAVOR :POSTGRES
2026-09-29 17:50:06,077 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:50:06,077 [I] Checking connection
2026-09-29 17:50:06,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 17:50:06,311 [I] connection success
2026-09-29 17:50:06,312 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 17:50:06,312 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 17:50:06,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 17:50:06,564 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 17:50:06,564 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:50:06,564 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 17:50:06,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.183.212:46993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 17:50:06,825 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 17:50:07.313987 3807 mini_ranger_kms.cc:310] Created kms service
I20260929 17:50:07.522253 3807 mini_ranger_kms.cc:326] Created kudu user
I20260929 17:50:07.564270 3807 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 17:50:07 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704205, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:07.961669 3807 mini_ranger_kms.cc:384] Added ranger policy
I20260929 17:50:07.961827 3807 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms:/tmp/dist-test-task6qiWXY/build/debug/bin/postgresql.jar:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6qiWXY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 17:50:07.961850 3807 mini_ranger_kms.cc:240] Using host: 127.3.183.212
I20260929 17:50:07.965425 3807 mini_ranger_kms.cc:276] Ranger KMS PID: 15980
I20260929 17:50:07.965505 3807 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.183.212:51963
Sep 29, 2026 5:50:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:50:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:50:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 5:50:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 5:50:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 5:50:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 17:50:09.950259 3807 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.857s user 0.011s sys 0.131s
I20260929 17:50:09.950398 3807 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 17:50:09.950466 3807 mini_ranger_kms.cc:401] 127.3.183.212:51963/kms/v1/keys
Sep 29 17:50:10 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704210, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.183.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 17:50:10 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704210, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.183.212@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
Sep 29 17:50:10 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704210, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.183.212@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
Sep 29 17:50:13 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:50:13.163353 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35585
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:35585
--ranger_config_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:13.272701 16031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:13.272951 16031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:13.272989 16031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:13.276916 16031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:13.276968 16031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:13.276988 16031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:13.277002 16031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:13.277015 16031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:13.282366 16031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.183.212:51963/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:35585
--ranger_config_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:13.283306 16031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:13.284307 16031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:13.292440 16036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:13.292461 16031 server_base.cc:1034] running on GCE node
W20260929 17:50:13.292411 16037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:13.292403 16039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:13.293087 16031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:13.294178 16031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:13.295339 16031 hybrid_clock.cc:648] HybridClock initialized: now 1790704213295327 us; error 24 us; skew 500 ppm
Sep 29 17:50:13 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704213, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:13.297992 16031 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:13.298799 16031 webserver.cc:460] Webserver started at http://127.3.183.254:45437/ using document root <none> and password file <none>
I20260929 17:50:13.299319 16031 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:13.299371 16031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:13.299553 16031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:50:13 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704213, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:13.364296 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "685ccc6cc11441ce932a23578688198e"
format_stamp: "Formatted at 2026-09-29 17:50:13 on dist-test-slave-wbpt"
server_key: "9519df6bef43dd08a9853a516fd83467"
server_key_iv: "860bf9432f74df9552610fe1fc4e2df3"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:13.364902 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "685ccc6cc11441ce932a23578688198e"
format_stamp: "Formatted at 2026-09-29 17:50:13 on dist-test-slave-wbpt"
server_key: "9519df6bef43dd08a9853a516fd83467"
server_key_iv: "860bf9432f74df9552610fe1fc4e2df3"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:13.368743 16031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:50:13.385522 16046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:13.387250 16031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:50:13.387415 16031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "685ccc6cc11441ce932a23578688198e"
format_stamp: "Formatted at 2026-09-29 17:50:13 on dist-test-slave-wbpt"
server_key: "9519df6bef43dd08a9853a516fd83467"
server_key_iv: "860bf9432f74df9552610fe1fc4e2df3"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:13.387563 16031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:13.408929 16031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:13.412356 16031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:13.412546 16031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:13.419610 16031 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:35585
I20260929 17:50:13.419653 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:35585 every 8 connection(s)
I20260929 17:50:13.420603 16031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:13.421144 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16031
I20260929 17:50:13.421265 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:50:13.421554 3807 external_mini_cluster.cc:1428] Setting key bf33f541c569f72283af107b45f21e4d
I20260929 17:50:13.423705 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:13.429105 16099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Bootstrap starting.
I20260929 17:50:13.431545 16099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:13.432523 16099 log.cc:824] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:13.434468 16099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: No bootstrap required, opened a new log
Sep 29 17:50:13 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:13.438661 16099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:13.438889 16099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:13.438966 16099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685ccc6cc11441ce932a23578688198e, State: Initialized, Role: FOLLOWER
I20260929 17:50:13.439499 16099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:13.439708 16099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:50:13.439822 16099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:50:13.439935 16099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:13.441035 16099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:13.441409 16099 leader_election.cc:302] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 685ccc6cc11441ce932a23578688198e; no voters:
I20260929 17:50:13.441781 16099 leader_election.cc:288] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:50:13.441875 16104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:13.442158 16104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 LEADER]: Becoming Leader. State: Replica: 685ccc6cc11441ce932a23578688198e, State: Running, Role: LEADER
I20260929 17:50:13.442504 16104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:13.443194 16099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:13.443761 16104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "685ccc6cc11441ce932a23578688198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } } }
I20260929 17:50:13.443974 16104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:13.444213 16104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 685ccc6cc11441ce932a23578688198e. Latest consensus state: current_term: 1 leader_uuid: "685ccc6cc11441ce932a23578688198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } } }
I20260929 17:50:13.444357 16104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:13.444901 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:13.423094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.3.183.254:35585)
0929 17:50:13.423560 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:13.423579 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:13.429628 (+ 6049us) server_negotiation.cc:407] Connection header received
0929 17:50:13.430604 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:13.430628 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:13.431083 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:13.431462 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:13.432372 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:13.433937 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:13.435165 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:13.435966 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:13.436125 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:13.439112 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:13.439118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:13.439136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:13.439167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:13.441082 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:13.441589 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:13.441594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:13.441600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:13.441728 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:13.442052 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:13.442055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:13.442057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:13.442537 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:13.443400 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:13.443549 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 17:50:13.443853 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260929 17:50:13.445333 16099 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 17:50:13.451854 16112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:13.455382 16112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:13.462109 16112 catalog_manager.cc:1347] Generated new cluster ID: 2d2eb27241334a8dbfc18398fda3eb7e
I20260929 17:50:13.462266 16112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:50:13.476308 16112 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:50:13.477789 16112 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:50:13.483112 16112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Generated new TSK 0
I20260929 17:50:13.484171 16112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:50:14.157194 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35585
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:14.319262 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:14.319743 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:14.319888 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:14.324642 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:14.324760 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:14.324882 16144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:50:14.333003 16144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.183.212:51963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:14.334743 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:14.336205 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:14.348531 16152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:14.348562 16154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:14.349056 16151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:14.349424 16144 server_base.cc:1034] running on GCE node
I20260929 17:50:14.349934 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:14.350726 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:14.351846 16144 hybrid_clock.cc:648] HybridClock initialized: now 1790704214351830 us; error 40 us; skew 500 ppm
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:14.354826 16144 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:50:14.355687 16144 webserver.cc:460] Webserver started at http://127.3.183.193:44947/ using document root <none> and password file <none>
I20260929 17:50:14.356243 16144 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:14.356309 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:14.356567 16144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:14.373476 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "92c36dc12264494289d2fe8a0d511b95"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "e1da95296677e35d0b0f0fdb493abaee"
server_key_iv: "e9ad62d0078826461c9530b9ba9bb030"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:14.376247 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "92c36dc12264494289d2fe8a0d511b95"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "e1da95296677e35d0b0f0fdb493abaee"
server_key_iv: "e9ad62d0078826461c9530b9ba9bb030"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:14.381382 16144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 17:50:14.412204 16161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:14.414326 16144 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.001s sys 0.000s
I20260929 17:50:14.414470 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "92c36dc12264494289d2fe8a0d511b95"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "e1da95296677e35d0b0f0fdb493abaee"
server_key_iv: "e9ad62d0078826461c9530b9ba9bb030"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:14.414621 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:14.455382 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:14.459372 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:14.459553 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:14.460160 16144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:14.461004 16144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:14.461053 16144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:14.461103 16144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:14.461144 16144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:14.470763 16144 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:34031
I20260929 17:50:14.470793 16276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:34031 every 8 connection(s)
I20260929 17:50:14.471761 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:50:14.479143 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16144
I20260929 17:50:14.479259 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:50:14.479562 3807 external_mini_cluster.cc:1428] Setting key cbf0bf034c5dc977212525f1631090c4
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:14.485984 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:14.473983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:59019 (local address 127.3.183.254:35585)
0929 17:50:14.474307 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:14.474315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:14.475001 (+ 686us) server_negotiation.cc:407] Connection header received
0929 17:50:14.475987 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:14.475990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:14.476054 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:14.476138 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:14.477193 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.477743 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.478871 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.479190 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.479269 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:14.482102 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:14.482106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:14.482108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:14.482132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:14.483669 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:14.484278 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:14.484282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:14.484283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:14.484373 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:14.484747 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:14.484750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:14.484752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:14.484976 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:14.485066 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:14.485690 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 17:50:14.485810 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":148,"threads_started":1}
I20260929 17:50:14.487825 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:14.474415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35585 (local address 127.3.183.193:59019)
0929 17:50:14.474858 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:14.474910 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:14.475763 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:14.476318 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:14.476325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:14.476588 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:14.477049 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:14.477062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.477854 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.477856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:14.478741 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:14.478751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.479471 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.479475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:14.479676 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:14.480491 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:14.480515 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:14.481921 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:14.483795 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:14.483801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:14.483818 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:14.484166 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:14.484472 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:14.484475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:14.484477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:14.484647 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:14.485130 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:14.485137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:14.485433 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 17:50:14.486367 (+ 934us) client_negotiation.cc:240] Negotiation successful
0929 17:50:14.486715 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260929 17:50:14.489672 16277 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35585
I20260929 17:50:14.489895 16277 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:14.490371 16277 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:14.492082 16063 ts_manager.cc:194] Registered new tserver with Master: 92c36dc12264494289d2fe8a0d511b95 (127.3.183.193:34031)
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:50:14.493600 16063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:59019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:50:14.515726 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35585
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:14.670601 16283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:14.671109 16283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:14.671231 16283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:14.677825 16283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:14.677917 16283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:14.678009 16283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:50:14.686182 16283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.183.212:51963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:14.687844 16283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:14.689215 16283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:14.701501 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:14.701717 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:14.702181 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:14.702291 16283 server_base.cc:1034] running on GCE node
I20260929 17:50:14.702917 16283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:14.703770 16283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:14.704998 16283 hybrid_clock.cc:648] HybridClock initialized: now 1790704214704971 us; error 43 us; skew 500 ppm
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:14.708671 16283 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:50:14.709846 16283 webserver.cc:460] Webserver started at http://127.3.183.194:36127/ using document root <none> and password file <none>
I20260929 17:50:14.710615 16283 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:14.710759 16283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:14.711030 16283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:14.727972 16283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "3c1ec99aaac64e299b0752ead119c9b7"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "d53928bff76379e9b27f753cdd4fdfe7"
server_key_iv: "08bdfb72829cc5bb5418698733247b96"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:14.729043 16283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "3c1ec99aaac64e299b0752ead119c9b7"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "d53928bff76379e9b27f753cdd4fdfe7"
server_key_iv: "08bdfb72829cc5bb5418698733247b96"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:14.734740 16283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 17:50:14.758636 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:14.760558 16283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:50:14.760710 16283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "3c1ec99aaac64e299b0752ead119c9b7"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "d53928bff76379e9b27f753cdd4fdfe7"
server_key_iv: "08bdfb72829cc5bb5418698733247b96"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:14.760854 16283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:14.794664 16283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:14.798436 16283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:14.798705 16283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:14.799469 16283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:14.800616 16283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:14.800726 16283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:14.800827 16283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:14.800894 16283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:14.810778 16283 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:35871
I20260929 17:50:14.811970 16283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:50:14.814788 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:35871 every 8 connection(s)
I20260929 17:50:14.820657 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16283
I20260929 17:50:14.820794 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:50:14.821072 3807 external_mini_cluster.cc:1428] Setting key ff130295dd4953c398555f16f765f5cd
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:14.829561 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:14.816535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:55225 (local address 127.3.183.254:35585)
0929 17:50:14.817865 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:14.817869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:14.817886 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:50:14.818505 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:14.818508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:14.818565 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:14.818636 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:14.819946 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.820595 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.821945 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.822257 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.822327 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:14.825396 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:14.825400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:14.825403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:14.825433 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:14.827067 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:14.827680 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:14.827683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:14.827685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:14.827775 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:14.828211 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:14.828215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:14.828217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:14.828448 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:14.828535 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:14.829235 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 17:50:14.829372 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239}
I20260929 17:50:14.831606 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:14.816919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35585 (local address 127.3.183.194:55225)
0929 17:50:14.817406 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:14.817458 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:14.818311 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:14.818798 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:14.818805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:14.819091 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:14.819744 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:14.819764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.820713 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.820716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:14.821824 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:14.821835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:14.822531 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:14.822535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:14.822731 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:14.823386 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:14.823409 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:14.825188 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:14.827207 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:14.827213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:14.827233 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:14.827568 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:14.827912 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:14.827915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:14.827917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:14.828101 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:14.828618 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:14.828625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:14.828953 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 17:50:14.830092 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0929 17:50:14.830465 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260929 17:50:14.833135 16413 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35585
I20260929 17:50:14.833376 16413 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:14.833859 16413 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:14.834887 16063 ts_manager.cc:194] Registered new tserver with Master: 3c1ec99aaac64e299b0752ead119c9b7 (127.3.183.194:35871)
I20260929 17:50:14.835570 16063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:55225
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:50:14.859597 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35585
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:50:14 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704214, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
W20260929 17:50:15.000701 16418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:15.001151 16418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:15.001243 16418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:15.005913 16418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:15.005973 16418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:15.006054 16418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:50:15.011714 16418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.183.212:51963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:15.012837 16418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:15.013864 16418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:15.023320 16424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:15.023489 16418 server_base.cc:1034] running on GCE node
W20260929 17:50:15.023324 16426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:15.023768 16423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:15.024273 16418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:15.024891 16418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:15.026109 16418 hybrid_clock.cc:648] HybridClock initialized: now 1790704215026049 us; error 71 us; skew 500 ppm
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:15.028820 16418 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:50:15.029704 16418 webserver.cc:460] Webserver started at http://127.3.183.195:33549/ using document root <none> and password file <none>
I20260929 17:50:15.030244 16418 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:15.030309 16418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:15.030525 16418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:15.055017 16418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "1d7438f68ad34a48a29433533cb049c1"
format_stamp: "Formatted at 2026-09-29 17:50:15 on dist-test-slave-wbpt"
server_key: "d68b1900366900871e065f03fd7684db"
server_key_iv: "4598f50851f269fa660d8947e4089daf"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:15.055811 16418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "1d7438f68ad34a48a29433533cb049c1"
format_stamp: "Formatted at 2026-09-29 17:50:15 on dist-test-slave-wbpt"
server_key: "d68b1900366900871e065f03fd7684db"
server_key_iv: "4598f50851f269fa660d8947e4089daf"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:15.061302 16418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 17:50:15.077872 16433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.079632 16418 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:50:15.080098 16418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "1d7438f68ad34a48a29433533cb049c1"
format_stamp: "Formatted at 2026-09-29 17:50:15 on dist-test-slave-wbpt"
server_key: "d68b1900366900871e065f03fd7684db"
server_key_iv: "4598f50851f269fa660d8947e4089daf"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:15.080297 16418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:15.104646 16418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:15.107525 16418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:15.107687 16418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:15.108285 16418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:15.109105 16418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:15.109151 16418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.109201 16418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:15.109254 16418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.119622 16418 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:37553
I20260929 17:50:15.120787 16418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:50:15.124568 16546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:37553 every 8 connection(s)
I20260929 17:50:15.127286 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16418
I20260929 17:50:15.127401 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:50:15.127688 3807 external_mini_cluster.cc:1428] Setting key fca1332a1c432aad342c7529d75caef1
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:15.139479 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:15.126173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:39637 (local address 127.3.183.254:35585)
0929 17:50:15.127721 (+ 1548us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:15.127726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:15.128333 (+ 607us) server_negotiation.cc:407] Connection header received
0929 17:50:15.129845 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:15.129848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:15.129907 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:15.129950 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:15.131515 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.132105 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.133311 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.133534 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.133616 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:15.136267 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:15.136270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:15.136272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:15.136295 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:15.137585 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:15.138055 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:15.138058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:15.138059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:15.138138 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:15.138455 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:15.138458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:15.138459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:15.138629 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:15.138709 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:15.139192 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 17:50:15.139318 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446}
I20260929 17:50:15.141388 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:15.126625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35585 (local address 127.3.183.195:39637)
0929 17:50:15.128188 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:15.128243 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:15.129623 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:15.130272 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:15.130279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:15.130650 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:15.131273 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:15.131288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.132220 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.132223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:15.133194 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:15.133204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.133665 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.133669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:15.133878 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:15.134463 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:15.134479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:15.136115 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:15.137688 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:15.137694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:15.137710 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:15.137965 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:15.138217 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:15.138220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:15.138222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:15.138356 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:15.138720 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:15.138727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:15.138966 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 17:50:15.139137 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 17:50:15.139459 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":179,"threads_started":1}
I20260929 17:50:15.142711 16547 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35585
I20260929 17:50:15.142972 16547 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:15.143462 16547 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:15.144533 16063 ts_manager.cc:194] Registered new tserver with Master: 1d7438f68ad34a48a29433533cb049c1 (127.3.183.195:37553)
I20260929 17:50:15.145190 16063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:39637
I20260929 17:50:15.153228 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:50:15.153861 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16144
I20260929 17:50:15.159412 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16283
I20260929 17:50:15.164536 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16418
I20260929 17:50:15.169227 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16031
I20260929 17:50:15.174814 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35585
--webserver_interface=127.3.183.254
--webserver_port=45437
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:35585
--ranger_config_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:15.281435 16550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:15.281699 16550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:15.281756 16550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:15.285764 16550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:15.285817 16550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:15.285843 16550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:15.285864 16550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:15.285893 16550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:15.291239 16550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:35585
--ranger_config_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=45437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:15.292268 16550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:15.293246 16550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:50:15.301327 16550 server_base.cc:1034] running on GCE node
W20260929 17:50:15.301294 16556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:15.301262 16555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:15.301281 16558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:15.302011 16550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:15.303171 16550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:15.304348 16550 hybrid_clock.cc:648] HybridClock initialized: now 1790704215304334 us; error 27 us; skew 500 ppm
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:15.306933 16550 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:15.307830 16550 webserver.cc:460] Webserver started at http://127.3.183.254:45437/ using document root <none> and password file <none>
I20260929 17:50:15.308344 16550 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:15.308413 16550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:15.354702 16550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:50:15.357234 16566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.358656 16550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:50:15.358794 16550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "685ccc6cc11441ce932a23578688198e"
format_stamp: "Formatted at 2026-09-29 17:50:13 on dist-test-slave-wbpt"
server_key: "9519df6bef43dd08a9853a516fd83467"
server_key_iv: "860bf9432f74df9552610fe1fc4e2df3"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:15.359355 16550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:15.374663 16550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:15.377274 16550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:15.377444 16550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:15.384120 16550 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:35585
I20260929 17:50:15.384179 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:35585 every 8 connection(s)
I20260929 17:50:15.385131 16550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:15.389365 16619 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:50:15.390969 16619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Bootstrap starting.
I20260929 17:50:15.391462 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16550
I20260929 17:50:15.391892 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:34031
--local_ip_for_outbound_sockets=127.3.183.193
--tserver_master_addrs=127.3.183.254:35585
--webserver_port=44947
--webserver_interface=127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:15.407086 16619 log.cc:824] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:15.413661 16619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:50:15.414006 16619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Bootstrap complete.
I20260929 17:50:15.417919 16619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:15.418313 16619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685ccc6cc11441ce932a23578688198e, State: Initialized, Role: FOLLOWER
I20260929 17:50:15.418783 16619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:15.418905 16619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:50:15.418972 16619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:50:15.419072 16619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:50:15.420944 16619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:15.421304 16619 leader_election.cc:302] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 685ccc6cc11441ce932a23578688198e; no voters:
I20260929 17:50:15.421594 16619 leader_election.cc:288] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:50:15.421731 16623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:50:15.421967 16623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [term 2 LEADER]: Becoming Leader. State: Replica: 685ccc6cc11441ce932a23578688198e, State: Running, Role: LEADER
I20260929 17:50:15.422312 16623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } }
I20260929 17:50:15.422694 16619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:15.423281 16623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 685ccc6cc11441ce932a23578688198e. Latest consensus state: current_term: 2 leader_uuid: "685ccc6cc11441ce932a23578688198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } } }
I20260929 17:50:15.423465 16623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:15.423666 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "685ccc6cc11441ce932a23578688198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685ccc6cc11441ce932a23578688198e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35585 } } }
I20260929 17:50:15.423851 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:15.423890 16631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:15.427169 16631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:15.427675 16619 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 17:50:15.427836 16631 catalog_manager.cc:1259] Loaded cluster ID: 2d2eb27241334a8dbfc18398fda3eb7e
I20260929 17:50:15.427881 16631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:50:15.430243 16631 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:50:15.430857 16631 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Loaded TSK: 0
I20260929 17:50:15.432063 16631 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:50:15.530871 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:15.531338 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:15.531447 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:15.537559 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:15.537721 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:15.537854 16621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:50:15.545734 16621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:34031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=44947
--tserver_master_addrs=127.3.183.254:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:15.547134 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:15.548573 16621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:15.560027 16657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:15.560070 16659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:15.560027 16656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:15.561039 16621 server_base.cc:1034] running on GCE node
I20260929 17:50:15.561440 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:15.562215 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:15.563458 16621 hybrid_clock.cc:648] HybridClock initialized: now 1790704215563420 us; error 55 us; skew 500 ppm
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:15.566656 16621 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:50:15.567909 16621 webserver.cc:460] Webserver started at http://127.3.183.193:44947/ using document root <none> and password file <none>
I20260929 17:50:15.568782 16621 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:15.568948 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:15.588902 16621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 17:50:15.592197 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.593935 16621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 17:50:15.594067 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "92c36dc12264494289d2fe8a0d511b95"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "e1da95296677e35d0b0f0fdb493abaee"
server_key_iv: "e9ad62d0078826461c9530b9ba9bb030"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:15.594767 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:15.614614 16621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:15.618135 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:15.618332 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:15.619076 16621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:15.620147 16621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:15.620213 16621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.620271 16621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:15.620301 16621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.630765 16621 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:34031
I20260929 17:50:15.630807 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:34031 every 8 connection(s)
I20260929 17:50:15.631742 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:50:15.639312 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16621
I20260929 17:50:15.639863 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:35871
--local_ip_for_outbound_sockets=127.3.183.194
--tserver_master_addrs=127.3.183.254:35585
--webserver_port=36127
--webserver_interface=127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:15.650897 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:15.634631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35585 (local address 127.3.183.193:60955)
0929 17:50:15.635537 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:15.635576 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:15.636443 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:15.637563 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:15.637571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:15.637887 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:15.638336 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:15.638350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.639952 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.639955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:15.641239 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:15.641252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.642493 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.642497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:15.642699 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:15.643292 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:15.643312 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:15.645069 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:15.647379 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:15.647385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:15.647400 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:15.647733 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:15.648178 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:15.648181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:15.648183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:15.648340 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:15.649089 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:15.649095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:15.649367 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 17:50:15.649553 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 17:50:15.649860 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":116,"threads_started":1}
I20260929 17:50:15.652765 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:15.634471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:60955 (local address 127.3.183.254:35585)
0929 17:50:15.636037 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:15.636057 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:15.636104 (+ 47us) server_negotiation.cc:407] Connection header received
0929 17:50:15.636847 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:15.636858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:15.637177 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:15.637468 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:15.638494 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.639772 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.641383 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.642118 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.642271 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:15.645291 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:15.645297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:15.645314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:15.645346 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:15.647250 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:15.647927 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:15.647933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:15.647939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:15.648077 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:15.648435 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:15.648438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:15.648440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:15.648910 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:15.649035 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:15.651461 (+ 2426us) server_negotiation.cc:299] Negotiation successful
0929 17:50:15.651797 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1407,"thread_start_us":110,"threads_started":1}
I20260929 17:50:15.656437 16781 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35585
I20260929 17:50:15.656692 16781 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:15.657284 16781 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
W20260929 17:50:15.659475 16781 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:35585 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 17:50:15.659595 16781 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:15.659744 16781 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:15.660601 16781 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:15.660749 16781 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
W20260929 17:50:15.661480 16781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:50:15.781477 16785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:15.781785 16785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:15.781847 16785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:15.787091 16785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:15.787165 16785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:15.787249 16785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:50:15.792929 16785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:35871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=36127
--tserver_master_addrs=127.3.183.254:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:15.794059 16785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:15.795090 16785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:15.805132 16790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:15.805336 16793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:15.805307 16785 server_base.cc:1034] running on GCE node
W20260929 17:50:15.805169 16791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:15.805879 16785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:15.806658 16785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:15.807857 16785 hybrid_clock.cc:648] HybridClock initialized: now 1790704215807833 us; error 31 us; skew 500 ppm
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:15.811296 16785 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:50:15.812438 16785 webserver.cc:460] Webserver started at http://127.3.183.194:36127/ using document root <none> and password file <none>
I20260929 17:50:15.813174 16785 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:15.813280 16785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:15.834836 16785 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260929 17:50:15.840986 16801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.842821 16785 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260929 17:50:15.842988 16785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "3c1ec99aaac64e299b0752ead119c9b7"
format_stamp: "Formatted at 2026-09-29 17:50:14 on dist-test-slave-wbpt"
server_key: "d53928bff76379e9b27f753cdd4fdfe7"
server_key_iv: "08bdfb72829cc5bb5418698733247b96"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:15.843770 16785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:15.869328 16785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:15.873101 16785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:15.873318 16785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:15.874099 16785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:15.875164 16785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:15.875234 16785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.875290 16785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:15.875316 16785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:15.887264 16785 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:35871
I20260929 17:50:15.887449 16914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:35871 every 8 connection(s)
I20260929 17:50:15.888581 16785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:50:15.894542 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16785
I20260929 17:50:15.895026 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.183.212:51963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:37553
--local_ip_for_outbound_sockets=127.3.183.195
--tserver_master_addrs=127.3.183.254:35585
--webserver_port=33549
--webserver_interface=127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:50:15 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:15.904412 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:15.890465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:34469 (local address 127.3.183.254:35585)
0929 17:50:15.890600 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:15.890604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:15.891665 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 17:50:15.892648 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:15.892651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:15.892714 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:15.892782 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:15.893932 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.894536 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.896119 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.896439 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.896534 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:15.900265 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:15.900269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:15.900272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:15.900302 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:15.901865 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:15.902511 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:15.902515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:15.902517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:15.902606 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:15.903020 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:15.903023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:15.903025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:15.903248 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:15.903330 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:15.904093 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 17:50:15.904237 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:50:15.906358 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:15.891063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35585 (local address 127.3.183.194:34469)
0929 17:50:15.891517 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:15.891567 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:15.892443 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:15.892965 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:15.892972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:15.893293 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:15.893788 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:15.893802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.894688 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.894691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:15.895985 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:15.895996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:15.896732 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:15.896736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:15.896933 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:15.897610 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:15.897630 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:15.900097 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:15.902010 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:15.902017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:15.902034 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:15.902392 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:15.902714 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:15.902717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:15.902719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:15.902899 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:15.903408 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:15.903415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:15.903758 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 17:50:15.904869 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0929 17:50:15.905243 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260929 17:50:15.907950 16915 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35585
I20260929 17:50:15.908206 16915 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:15.908731 16915 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
W20260929 17:50:15.910516 16915 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:35585 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 17:50:15.910624 16915 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:15.910768 16915 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:15.911511 16915 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:15.911659 16915 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
W20260929 17:50:15.912369 16915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:50:16.026130 16918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:16.026412 16918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:16.026473 16918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:16.030612 16918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:16.030675 16918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:16.030751 16918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:50:16.037648 16918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.183.212:51963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:37553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=33549
--tserver_master_addrs=127.3.183.254:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:16.039024 16918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:16.040369 16918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:16.052911 16925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:16.052994 16927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:16.053288 16924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:16.053403 16918 server_base.cc:1034] running on GCE node
I20260929 17:50:16.054019 16918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:16.054853 16918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:16.056053 16918 hybrid_clock.cc:648] HybridClock initialized: now 1790704216056026 us; error 41 us; skew 500 ppm
Sep 29 17:50:16 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:16.059645 16918 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:50:16.060853 16918 webserver.cc:460] Webserver started at http://127.3.183.195:33549/ using document root <none> and password file <none>
I20260929 17:50:16.061650 16918 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:16.061784 16918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:50:16 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:16.084350 16918 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:50:16.086803 16935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:16.088248 16918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:50:16.088431 16918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "1d7438f68ad34a48a29433533cb049c1"
format_stamp: "Formatted at 2026-09-29 17:50:15 on dist-test-slave-wbpt"
server_key: "d68b1900366900871e065f03fd7684db"
server_key_iv: "4598f50851f269fa660d8947e4089daf"
server_key_version: "kuduclusterkey@0"
I20260929 17:50:16.089262 16918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:16.120852 16918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:16.124692 16918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:16.124900 16918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:16.125646 16918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:16.126694 16918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:16.126765 16918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:16.126825 16918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:16.126857 16918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:16.144132 16918 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:37553
I20260929 17:50:16.144325 17048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:37553 every 8 connection(s)
I20260929 17:50:16.145334 16918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:50:16.155427 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 16918
Sep 29 17:50:16 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:16.163729 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:16.147368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:36223 (local address 127.3.183.254:35585)
0929 17:50:16.147509 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:16.147514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:16.148553 (+ 1039us) server_negotiation.cc:407] Connection header received
0929 17:50:16.149852 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:16.149855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:16.149922 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:16.150014 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:16.151098 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:16.151662 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:16.153346 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:16.153682 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:16.153807 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:16.157875 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:16.157879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:16.157882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:16.157912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:16.159625 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:16.160315 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:16.160319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:16.160321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:16.160411 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:16.161977 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:16.161981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:16.161983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:16.162201 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:16.162301 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:16.163407 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0929 17:50:16.163542 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:50:16.164775 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:16.147848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:35585 (local address 127.3.183.195:36223)
0929 17:50:16.148397 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:16.148449 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:16.149641 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:16.150196 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:16.150205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:16.150480 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:16.150954 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:16.150967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:16.151793 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:16.151798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:16.153191 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:16.153202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:16.153900 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:16.153906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:16.154118 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:16.154819 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:16.155878 (+ 1059us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:16.157658 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:16.159755 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:16.159762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:16.159805 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:16.160185 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:16.160514 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:16.160517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:16.160519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:16.160701 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:16.162714 (+ 2013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:16.162722 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:16.163091 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 17:50:16.163313 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 17:50:16.163681 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":113,"threads_started":1}
I20260929 17:50:16.166179 17049 heartbeater.cc:347] Connected to a master server at 127.3.183.254:35585
I20260929 17:50:16.166441 17049 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:16.166944 17049 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:16.168495 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:16.156140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.3.183.254:35585)
0929 17:50:16.158133 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:16.158141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:16.158155 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:50:16.158189 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:16.158192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:16.158238 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:16.158319 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:16.159856 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:16.160698 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:16.161922 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:16.162231 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:16.162310 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:16.163982 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:16.163985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:16.163988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:16.164009 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:16.165754 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:16.166135 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:16.166139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:16.166141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:16.166227 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:16.167594 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:16.167598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:16.167600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:16.167843 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:16.168031 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:16.168205 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:50:16.168319 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1921,"thread_start_us":80,"threads_started":1}
W20260929 17:50:16.168534 17049 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:35585 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 17:50:16.168649 17049 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:16.168833 17049 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:16.169726 17049 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:16.169904 17049 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
W20260929 17:50:16.170706 17049 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 17:50:16 dist-test-slave-wbpt krb5kdc[15587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:16.661712 16781 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:16.662027 16781 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:16.663379 16583 ts_manager.cc:194] Registered new tserver with Master: 92c36dc12264494289d2fe8a0d511b95 (127.3.183.193:34031)
I20260929 17:50:16.665480 16583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:60955
I20260929 17:50:16.912609 16915 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:16.913026 16915 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
Sep 29 17:50:16 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:16.914019 16583 ts_manager.cc:194] Registered new tserver with Master: 3c1ec99aaac64e299b0752ead119c9b7 (127.3.183.194:35871)
I20260929 17:50:16.914891 16583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:34469
I20260929 17:50:17.170928 17049 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:17.171241 17049 heartbeater.cc:511] Master 127.3.183.254:35585 requested a full tablet report, sending...
I20260929 17:50:17.172027 16583 ts_manager.cc:194] Registered new tserver with Master: 1d7438f68ad34a48a29433533cb049c1 (127.3.183.195:37553)
I20260929 17:50:17.172696 16583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:36223
I20260929 17:50:17.174011 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:50:17.185391 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.176067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.3.183.254:35585)
0929 17:50:17.176553 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:17.176558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:17.176841 (+ 283us) server_negotiation.cc:407] Connection header received
0929 17:50:17.177056 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:17.177059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:17.177138 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:17.177203 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:17.178187 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.179031 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.180209 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.180432 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.180531 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.181897 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:17.181902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:17.181905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:17.181932 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:17.183592 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:17.184037 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:17.184040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:17.184042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:17.184128 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:17.184405 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:17.184409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:17.184410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:17.184620 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:17.184759 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:17.185062 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 17:50:17.185205 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":88,"threads_started":1}
I20260929 17:50:17.190320 16583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:50:17.197330 16583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 17:50:17 dist-test-slave-wbpt krb5kdc[15587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for HTTP/127.3.183.212@KRBTEST.COM
I20260929 17:50:17.224553 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.209387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:34031 (local address 127.0.0.1:33268)
0929 17:50:17.218336 (+ 8949us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:17.218361 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:17.218501 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:17.219080 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:17.219087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:17.219118 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:17.219290 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.219299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.220880 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.220884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.222426 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.222438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.223865 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.223869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.224070 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.224123 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 17:50:17.224230 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 17:50:17.224383 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8867,"thread_start_us":106,"threads_started":1}
I20260929 17:50:17.225600 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.210626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.3.183.193:34031)
0929 17:50:17.216616 (+ 5990us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:17.216625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:17.218507 (+ 1882us) server_negotiation.cc:407] Connection header received
0929 17:50:17.218613 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:17.218619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:17.218876 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:17.219053 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:17.219444 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.220736 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.222583 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.223636 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.225060 (+ 1424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.225239 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:17.225297 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 17:50:17.225410 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5847,"thread_start_us":153,"threads_started":1}
I20260929 17:50:17.229426 16715 tablet_service.cc:1467] Processing CreateTablet for tablet d7800de14be847ce9574f5da11325a56 (DEFAULT_TABLE table=test-table [id=f86d181b50cf4c3d827ac5faabadb431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:17.230814 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7800de14be847ce9574f5da11325a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:17.236470 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.211531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:35871 (local address 127.0.0.1:54126)
0929 17:50:17.220097 (+ 8566us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:17.220112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:17.220335 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:17.221003 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:17.221006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:17.221025 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:17.221131 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.221136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.224626 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.224630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.225910 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.225922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.235949 (+ 10027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.235954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.236073 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.236101 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 17:50:17.236180 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 17:50:17.236278 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8497,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260929 17:50:17.239694 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.211961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.3.183.194:35871)
0929 17:50:17.217800 (+ 5839us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:17.217809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:17.220259 (+ 2450us) server_negotiation.cc:407] Connection header received
0929 17:50:17.220481 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:17.220487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:17.220687 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:17.220808 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:17.223197 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.224476 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.226076 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.234071 (+ 7995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.234192 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.234357 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:17.239360 (+ 5003us) server_negotiation.cc:299] Negotiation successful
0929 17:50:17.239502 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":79,"threads_started":1}
I20260929 17:50:17.240458 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.210242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.3.183.195:37553)
0929 17:50:17.219388 (+ 9146us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:17.219395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:17.219418 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:50:17.219490 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:17.219495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:17.219701 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:17.221552 (+ 1851us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:17.221793 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.223046 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.238830 (+ 15784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.239838 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.239945 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.240100 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:17.240184 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 17:50:17.240283 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9033,"thread_start_us":80,"threads_started":1}
I20260929 17:50:17.241348 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.209320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:37553 (local address 127.0.0.1:52754)
0929 17:50:17.212502 (+ 3182us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:17.212561 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:17.217909 (+ 5348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:17.221275 (+ 3366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:17.221279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:17.221297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:17.221384 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.221390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.237474 (+ 16084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.237478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.238644 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.238659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.239958 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.239962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.240052 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.240075 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:50:17.241120 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0929 17:50:17.241212 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2984,"thread_start_us":129,"threads_started":1}
I20260929 17:50:17.243381 16849 tablet_service.cc:1467] Processing CreateTablet for tablet d7800de14be847ce9574f5da11325a56 (DEFAULT_TABLE table=test-table [id=f86d181b50cf4c3d827ac5faabadb431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:17.244563 16849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7800de14be847ce9574f5da11325a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:17.244791 17083 tablet_bootstrap.cc:492] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: Bootstrap starting.
I20260929 17:50:17.247812 17083 tablet_bootstrap.cc:654] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:17.249370 17083 log.cc:824] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:17.250914 16981 tablet_service.cc:1467] Processing CreateTablet for tablet d7800de14be847ce9574f5da11325a56 (DEFAULT_TABLE table=test-table [id=f86d181b50cf4c3d827ac5faabadb431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:17.252048 16981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7800de14be847ce9574f5da11325a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:17.257956 17083 tablet_bootstrap.cc:492] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: No bootstrap required, opened a new log
I20260929 17:50:17.258271 17083 ts_tablet_manager.cc:1397] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260929 17:50:17.262387 17086 tablet_bootstrap.cc:492] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Bootstrap starting.
I20260929 17:50:17.264941 17083 raft_consensus.cc:348] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.265268 17083 raft_consensus.cc:374] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:17.265370 17083 raft_consensus.cc:729] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c36dc12264494289d2fe8a0d511b95, State: Initialized, Role: FOLLOWER
I20260929 17:50:17.265910 17083 consensus_queue.cc:260] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.267375 17083 ts_tablet_manager.cc:1428] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: Time spent starting tablet: real 0.009s user 0.003s sys 0.005s
I20260929 17:50:17.267814 16781 heartbeater.cc:503] Master 127.3.183.254:35585 was elected leader, sending a full tablet report...
I20260929 17:50:17.269937 17086 tablet_bootstrap.cc:654] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:17.271142 17086 log.cc:824] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:17.273649 17086 tablet_bootstrap.cc:492] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: No bootstrap required, opened a new log
I20260929 17:50:17.273860 17086 ts_tablet_manager.cc:1397] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260929 17:50:17.275245 17085 tablet_bootstrap.cc:492] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Bootstrap starting.
I20260929 17:50:17.278450 17085 tablet_bootstrap.cc:654] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:17.279860 17085 log.cc:824] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:17.282562 17086 raft_consensus.cc:348] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.282922 17086 raft_consensus.cc:374] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:17.283015 17086 raft_consensus.cc:729] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7438f68ad34a48a29433533cb049c1, State: Initialized, Role: FOLLOWER
I20260929 17:50:17.283542 17086 consensus_queue.cc:260] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.284070 17085 tablet_bootstrap.cc:492] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: No bootstrap required, opened a new log
I20260929 17:50:17.284335 17085 ts_tablet_manager.cc:1397] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 17:50:17.284889 17086 ts_tablet_manager.cc:1428] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260929 17:50:17.285319 17049 heartbeater.cc:503] Master 127.3.183.254:35585 was elected leader, sending a full tablet report...
I20260929 17:50:17.294137 17085 raft_consensus.cc:348] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.294448 17085 raft_consensus.cc:374] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:17.294531 17085 raft_consensus.cc:729] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1ec99aaac64e299b0752ead119c9b7, State: Initialized, Role: FOLLOWER
I20260929 17:50:17.295121 17085 consensus_queue.cc:260] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.296144 16915 heartbeater.cc:503] Master 127.3.183.254:35585 was elected leader, sending a full tablet report...
I20260929 17:50:17.296577 17085 ts_tablet_manager.cc:1428] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
W20260929 17:50:17.384186 16782 tablet.cc:2369] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:50:17.390949 16916 tablet.cc:2369] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:50:17.398243 17050 tablet.cc:2369] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:50:17.596318 17087 raft_consensus.cc:482] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:50:17.596638 17087 raft_consensus.cc:504] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.598167 17087 leader_election.cc:288] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d7438f68ad34a48a29433533cb049c1 (127.3.183.195:37553), 3c1ec99aaac64e299b0752ead119c9b7 (127.3.183.194:35871)
I20260929 17:50:17.604741 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.599053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:43391 (local address 127.3.183.194:35871)
0929 17:50:17.599179 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:17.599183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:17.599581 (+ 398us) server_negotiation.cc:407] Connection header received
0929 17:50:17.599812 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:17.599816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:17.599882 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:17.599937 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:17.600568 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.601318 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.603308 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.603941 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.604035 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.604085 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:17.604528 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 17:50:17.604611 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:50:17.604839 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.598971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:50895 (local address 127.3.183.195:37553)
0929 17:50:17.599120 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:17.599124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:17.599436 (+ 312us) server_negotiation.cc:407] Connection header received
0929 17:50:17.599659 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:17.599663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:17.599715 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:17.599802 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:17.600497 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.601309 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.603201 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.603842 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.603934 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.603987 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:17.604638 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 17:50:17.604717 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 17:50:17.605392 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.598392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:37553 (local address 127.3.183.193:50895)
0929 17:50:17.599322 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:17.599337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:17.599525 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:17.599974 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:17.599977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:17.599994 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:17.600342 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.600351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.601445 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.601449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.603066 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.603076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.604292 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.604296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.604494 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.604519 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:50:17.605024 (+ 505us) client_negotiation.cc:240] Negotiation successful
0929 17:50:17.605102 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":57,"thread_start_us":191,"threads_started":1}
I20260929 17:50:17.605602 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.598515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:35871 (local address 127.3.183.193:43391)
0929 17:50:17.599473 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:17.599487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:17.599656 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:17.600111 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:17.600114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:17.600144 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:17.600423 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.600430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.601467 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.601469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.603162 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:17.603173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.604177 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.604180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:17.604383 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.604408 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:50:17.605206 (+ 798us) client_negotiation.cc:240] Negotiation successful
0929 17:50:17.605281 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":33,"thread_start_us":155,"threads_started":1}
I20260929 17:50:17.606383 17001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7800de14be847ce9574f5da11325a56" candidate_uuid: "92c36dc12264494289d2fe8a0d511b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7438f68ad34a48a29433533cb049c1" is_pre_election: true
I20260929 17:50:17.606472 16869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7800de14be847ce9574f5da11325a56" candidate_uuid: "92c36dc12264494289d2fe8a0d511b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c1ec99aaac64e299b0752ead119c9b7" is_pre_election: true
I20260929 17:50:17.606710 17001 raft_consensus.cc:2454] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c36dc12264494289d2fe8a0d511b95 in term 0.
I20260929 17:50:17.606755 16869 raft_consensus.cc:2454] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c36dc12264494289d2fe8a0d511b95 in term 0.
I20260929 17:50:17.607280 16671 leader_election.cc:302] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7438f68ad34a48a29433533cb049c1, 92c36dc12264494289d2fe8a0d511b95; no voters:
I20260929 17:50:17.607755 17087 raft_consensus.cc:2781] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:50:17.607873 17087 raft_consensus.cc:482] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:50:17.607921 17087 raft_consensus.cc:3037] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:17.609514 17087 raft_consensus.cc:504] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.609894 17087 leader_election.cc:288] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [CANDIDATE]: Term 1 election: Requested vote from peers 1d7438f68ad34a48a29433533cb049c1 (127.3.183.195:37553), 3c1ec99aaac64e299b0752ead119c9b7 (127.3.183.194:35871)
I20260929 17:50:17.610371 17001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7800de14be847ce9574f5da11325a56" candidate_uuid: "92c36dc12264494289d2fe8a0d511b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7438f68ad34a48a29433533cb049c1"
I20260929 17:50:17.610507 17001 raft_consensus.cc:3037] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:17.610564 16869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7800de14be847ce9574f5da11325a56" candidate_uuid: "92c36dc12264494289d2fe8a0d511b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c1ec99aaac64e299b0752ead119c9b7"
I20260929 17:50:17.610683 16869 raft_consensus.cc:3037] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:17.612177 17001 raft_consensus.cc:2454] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c36dc12264494289d2fe8a0d511b95 in term 1.
I20260929 17:50:17.612434 16869 raft_consensus.cc:2454] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c36dc12264494289d2fe8a0d511b95 in term 1.
I20260929 17:50:17.612598 16671 leader_election.cc:302] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7438f68ad34a48a29433533cb049c1, 92c36dc12264494289d2fe8a0d511b95; no voters:
I20260929 17:50:17.612819 17087 raft_consensus.cc:2781] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:17.613044 17087 raft_consensus.cc:686] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 1 LEADER]: Becoming Leader. State: Replica: 92c36dc12264494289d2fe8a0d511b95, State: Running, Role: LEADER
I20260929 17:50:17.613385 17087 consensus_queue.cc:237] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } }
I20260929 17:50:17.616959 16583 catalog_manager.cc:5495] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92c36dc12264494289d2fe8a0d511b95 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "92c36dc12264494289d2fe8a0d511b95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92c36dc12264494289d2fe8a0d511b95" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 34031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 } health_report { overall_health: UNKNOWN } } }
I20260929 17:50:17.681219 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.676511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.3.183.193:34031)
0929 17:50:17.676641 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:17.676646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:17.676660 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:50:17.676804 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:17.676807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:17.676885 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:17.677027 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:50:17.677281 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.678070 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.679339 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:17.679657 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:17.679757 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:17.680121 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:50:17.680696 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:50:17.680820 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:17.680940 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 17:50:17.681007 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:50:17.688694 16869 raft_consensus.cc:1261] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 1 FOLLOWER]: Refusing update from remote peer 92c36dc12264494289d2fe8a0d511b95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:17.688925 17001 raft_consensus.cc:1261] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 1 FOLLOWER]: Refusing update from remote peer 92c36dc12264494289d2fe8a0d511b95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:17.689637 17102 consensus_queue.cc:1035] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7438f68ad34a48a29433533cb049c1" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:17.689821 17087 consensus_queue.cc:1035] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1ec99aaac64e299b0752ead119c9b7" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:17.698282 17105 mvcc.cc:205] Tried to move back new op lower bound from 7334724475644489728 to 7334724475347263488. Current Snapshot: MvccSnapshot[applied={T|T < 7334724475644489728}]
I20260929 17:50:17.699151 17106 mvcc.cc:205] Tried to move back new op lower bound from 7334724475644489728 to 7334724475347263488. Current Snapshot: MvccSnapshot[applied={T|T < 7334724475644489728}]
I20260929 17:50:17.703095 17107 mvcc.cc:205] Tried to move back new op lower bound from 7334724475644489728 to 7334724475347263488. Current Snapshot: MvccSnapshot[applied={T|T < 7334724475644489728}]
I20260929 17:50:17.709946 16583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:17.710121 16583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:17.712518 16583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 685ccc6cc11441ce932a23578688198e: Sending DeleteTablet for 3 replicas of tablet d7800de14be847ce9574f5da11325a56
I20260929 17:50:17.713637 16849 tablet_service.cc:1514] Processing DeleteTablet for tablet d7800de14be847ce9574f5da11325a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:17 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:54126
I20260929 17:50:17.713625 16981 tablet_service.cc:1514] Processing DeleteTablet for tablet d7800de14be847ce9574f5da11325a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:17 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:52754
I20260929 17:50:17.714293 16715 tablet_service.cc:1514] Processing DeleteTablet for tablet d7800de14be847ce9574f5da11325a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:17 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:33268
I20260929 17:50:17.714818 17117 tablet_replica.cc:330] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95: stopping tablet replica
I20260929 17:50:17.715185 17117 raft_consensus.cc:2229] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:50:17.715647 17117 raft_consensus.cc:2258] T d7800de14be847ce9574f5da11325a56 P 92c36dc12264494289d2fe8a0d511b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:17.715806 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16621
I20260929 17:50:17.717680 17116 tablet_replica.cc:330] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: stopping tablet replica
I20260929 17:50:17.718086 17116 raft_consensus.cc:2229] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:50:17.718485 17116 raft_consensus.cc:2258] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:17.719022 17115 tablet_replica.cc:330] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: stopping tablet replica
I20260929 17:50:17.719328 17115 raft_consensus.cc:2229] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:50:17.719663 17115 raft_consensus.cc:2258] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:17.721128 17116 ts_tablet_manager.cc:1905] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:50:17.721453 17115 ts_tablet_manager.cc:1905] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:50:17.723752 16568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:34031 (error 108)
I20260929 17:50:17.724201 17116 ts_tablet_manager.cc:1918] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:50:17.724289 17116 log.cc:1196] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/d7800de14be847ce9574f5da11325a56
I20260929 17:50:17.724598 17116 ts_tablet_manager.cc:1939] T d7800de14be847ce9574f5da11325a56 P 3c1ec99aaac64e299b0752ead119c9b7: Deleting consensus metadata
I20260929 17:50:17.724611 17115 ts_tablet_manager.cc:1918] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:50:17.724690 17115 log.cc:1196] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/d7800de14be847ce9574f5da11325a56
I20260929 17:50:17.724679 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16785
I20260929 17:50:17.724949 17115 ts_tablet_manager.cc:1939] T d7800de14be847ce9574f5da11325a56 P 1d7438f68ad34a48a29433533cb049c1: Deleting consensus metadata
I20260929 17:50:17.727054 16570 catalog_manager.cc:4841] TS 1d7438f68ad34a48a29433533cb049c1 (127.3.183.195:37553): tablet d7800de14be847ce9574f5da11325a56 (table test-table [id=f86d181b50cf4c3d827ac5faabadb431]) successfully deleted
I20260929 17:50:17.728310 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.727998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:34031 (local address 127.0.0.1:33298)
0929 17:50:17.728130 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:17.728216 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:34031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:50:17.728472 16568 catalog_manager.cc:4584] TS 92c36dc12264494289d2fe8a0d511b95 (127.3.183.193:34031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7800de14be847ce9574f5da11325a56: Network error: Client connection negotiation failed: client connection to 127.3.183.193:34031: connect: Connection refused (error 111)
I20260929 17:50:17.729773 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16918
I20260929 17:50:17.730185 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:17.729915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:35871 (local address 127.0.0.1:54136)
0929 17:50:17.730072 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:17.730117 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 17:50:17.736093 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 16550
I20260929 17:50:17.742044 3807 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 17:50:18.076934 3807 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 17:50:18.077042 3807 mini_ranger.cc:66] Stopping Ranger...
I20260929 17:50:18.411669 3807 mini_ranger.cc:68] Stopped Ranger
2026-09-29 17:50:18.411 UTC [15617] LOG: received smart shutdown request
2026-09-29 17:50:18.415 UTC [15617] LOG: background worker "logical replication launcher" (PID 15626) exited with exit code 1
2026-09-29 17:50:18.416 UTC [15621] LOG: shutting down
2026-09-29 17:50:18.590 UTC [15617] LOG: database system is shut down
2026-09-29T17:50:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48528 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:50:18 dist-test-slave-wbpt krb5kdc[17127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:50:18 dist-test-slave-wbpt krb5kdc[17127](info): set up 2 sockets
Sep 29 17:50:18 dist-test-slave-wbpt krb5kdc[17127](info): commencing operation
krb5kdc: starting...
W20260929 17:50:19.647562 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:50:19 dist-test-slave-wbpt krb5kdc[17127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:50:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:50:19.771078 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41775
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:41775,127.3.183.253:37045,127.3.183.252:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:19.880542 17140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:19.880797 17140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:19.880837 17140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:19.884778 17140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:19.884836 17140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:19.884855 17140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:19.884869 17140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:19.884883 17140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:19.890107 17140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:41775,127.3.183.253:37045,127.3.183.252:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:19.891129 17140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:19.892211 17140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:50:19.900386 17140 server_base.cc:1034] running on GCE node
W20260929 17:50:19.900346 17148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:19.900334 17145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:19.900331 17146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:19.901064 17140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:19.902165 17140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:19.903335 17140 hybrid_clock.cc:648] HybridClock initialized: now 1790704219903320 us; error 31 us; skew 500 ppm
Sep 29 17:50:19 dist-test-slave-wbpt krb5kdc[17127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:19.913723 17140 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:19.914928 17140 webserver.cc:460] Webserver started at http://127.3.183.254:40893/ using document root <none> and password file <none>
I20260929 17:50:19.915478 17140 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:19.915537 17140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:19.915705 17140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:19.917780 17140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "121dc28ce3ef40bfb306a4f7e51369ae"
format_stamp: "Formatted at 2026-09-29 17:50:19 on dist-test-slave-wbpt"
server_key: "aca176e82ab9c9c4919d29b060be17a5"
server_key_iv: "6c36265a614eb966282c8092f4addabc"
server_key_version: "encryptionkey@0"
I20260929 17:50:19.918296 17140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "121dc28ce3ef40bfb306a4f7e51369ae"
format_stamp: "Formatted at 2026-09-29 17:50:19 on dist-test-slave-wbpt"
server_key: "aca176e82ab9c9c4919d29b060be17a5"
server_key_iv: "6c36265a614eb966282c8092f4addabc"
server_key_version: "encryptionkey@0"
I20260929 17:50:19.921947 17140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:50:19.924261 17155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:19.925462 17140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:19.925562 17140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "121dc28ce3ef40bfb306a4f7e51369ae"
format_stamp: "Formatted at 2026-09-29 17:50:19 on dist-test-slave-wbpt"
server_key: "aca176e82ab9c9c4919d29b060be17a5"
server_key_iv: "6c36265a614eb966282c8092f4addabc"
server_key_version: "encryptionkey@0"
I20260929 17:50:19.925645 17140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:19.943460 17140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:19.946468 17140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:19.946622 17140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:19.953413 17140 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:41775
I20260929 17:50:19.953442 17207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:41775 every 8 connection(s)
I20260929 17:50:19.954355 17140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:19.957219 17208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:19.957366 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17140
I20260929 17:50:19.957526 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:50:19.957791 3807 external_mini_cluster.cc:1428] Setting key 868b5cc20093e3eebbb7039a4a943d8f
I20260929 17:50:19.961670 17208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.183.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.253@KRBTEST.COM" created.
Sep 29 17:50:19 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:19.976020 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:19.964273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41775 (local address 127.0.0.1:58574)
0929 17:50:19.964820 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:19.964860 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:19.965677 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:19.966345 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:19.966350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:19.966747 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:19.967034 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:19.967047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:19.968203 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:19.968205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:19.969005 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:19.969015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:19.969602 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:19.969606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:19.969697 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:19.970195 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:19.970211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:19.971516 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:19.973266 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:19.973273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:19.973283 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:19.973488 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:19.973769 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:19.973772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:19.973774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:19.973875 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:19.974405 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:19.974410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:19.974553 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 17:50:19.974696 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 17:50:19.975059 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":86,"thread_start_us":163,"threads_started":1}
I20260929 17:50:19.976022 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:19.965337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.3.183.254:41775)
0929 17:50:19.965690 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:19.965700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:19.965785 (+ 85us) server_negotiation.cc:407] Connection header received
0929 17:50:19.965994 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:19.965998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:19.966172 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:19.966328 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:19.967164 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:19.968092 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:19.969106 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:19.969469 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:19.969605 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:19.971640 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:19.971645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:19.971655 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:19.971674 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:19.973172 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:19.973581 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:19.973586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:19.973588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:19.973683 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:19.973962 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:19.973965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:19.973967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:19.974303 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:19.974417 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:19.974768 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 17:50:19.974976 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":100,"threads_started":1}
I20260929 17:50:19.978566 17208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:50:19.979220 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:19.978972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37314)
0929 17:50:19.979075 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:19.979154 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 17:50:19.979476 17156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
I20260929 17:50:19.980278 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:19.980084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37326)
0929 17:50:19.980174 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:19.980222 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260929 17:50:19.980461 17208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:37045: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab.
Entry for principal kudu/127.3.183.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab.
I20260929 17:50:19.983942 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:37045
--webserver_interface=127.3.183.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab
--principal=kudu/127.3.183.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:41775,127.3.183.253:37045,127.3.183.252:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:20.029716 17208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } attempt: 1
I20260929 17:50:20.030606 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.030283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37330)
0929 17:50:20.030448 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.030523 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 17:50:20.031363 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.031134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37332)
0929 17:50:20.031242 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.031293 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 17:50:20.031486 17208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:37045: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
W20260929 17:50:20.092384 17215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:20.092643 17215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:20.092686 17215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:20.096761 17215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:20.096832 17215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:20.096865 17215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:20.096887 17215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:20.096915 17215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:20.099649 17208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } attempt: 2
I20260929 17:50:20.100461 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.100183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37336)
0929 17:50:20.100325 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.100385 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 17:50:20.101200 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.100965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37344)
0929 17:50:20.101092 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.101140 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 17:50:20.101316 17208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:37045: Network error: Client connection negotiation failed: client connection to 127.3.183.253:37045: connect: Connection refused (error 111)
I20260929 17:50:20.102515 17215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:41775,127.3.183.253:37045,127.3.183.252:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab
--principal=kudu/127.3.183.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:37045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.183.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:20.103564 17215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:20.104578 17215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:50:20.112722 17215 server_base.cc:1034] running on GCE node
W20260929 17:50:20.112705 17221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:20.112686 17220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:20.112686 17223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:20.113488 17215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:20.114545 17215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:20.115729 17215 hybrid_clock.cc:648] HybridClock initialized: now 1790704220115711 us; error 28 us; skew 500 ppm
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:20.118290 17215 init.cc:377] Logged in from keytab as kudu/127.3.183.253@KRBTEST.COM (short username kudu)
I20260929 17:50:20.119161 17215 webserver.cc:460] Webserver started at http://127.3.183.253:42643/ using document root <none> and password file <none>
I20260929 17:50:20.119692 17215 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:20.119756 17215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:20.119969 17215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:20.121920 17215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data/instance:
uuid: "fc7a18fe43de4823a92f88e35cf868cc"
format_stamp: "Formatted at 2026-09-29 17:50:20 on dist-test-slave-wbpt"
server_key: "3fa471f44298bdf2c885a0a524bba085"
server_key_iv: "bd8fb2d02e59c34842766b931339ebc9"
server_key_version: "encryptionkey@0"
I20260929 17:50:20.122542 17215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/wal/instance:
uuid: "fc7a18fe43de4823a92f88e35cf868cc"
format_stamp: "Formatted at 2026-09-29 17:50:20 on dist-test-slave-wbpt"
server_key: "3fa471f44298bdf2c885a0a524bba085"
server_key_iv: "bd8fb2d02e59c34842766b931339ebc9"
server_key_version: "encryptionkey@0"
I20260929 17:50:20.126055 17215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:50:20.128333 17230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:20.129545 17215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:20.129649 17215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/wal
uuid: "fc7a18fe43de4823a92f88e35cf868cc"
format_stamp: "Formatted at 2026-09-29 17:50:20 on dist-test-slave-wbpt"
server_key: "3fa471f44298bdf2c885a0a524bba085"
server_key_iv: "bd8fb2d02e59c34842766b931339ebc9"
server_key_version: "encryptionkey@0"
I20260929 17:50:20.129746 17215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:20.145076 17215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:20.147859 17215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:20.148039 17215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:20.154822 17215 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.253:37045
I20260929 17:50:20.154843 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.253:37045 every 8 connection(s)
I20260929 17:50:20.155742 17215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
I20260929 17:50:20.158432 17283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:20.160027 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17215
I20260929 17:50:20.160157 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-1/wal/instance
I20260929 17:50:20.160447 3807 external_mini_cluster.cc:1428] Setting key 158e5bde68b297d8e2af8a8f0e918aaf
I20260929 17:50:20.162763 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.183.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.252@KRBTEST.COM" created.
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:20.176642 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.164696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.3.183.254:41775)
0929 17:50:20.164861 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.164866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.165679 (+ 813us) server_negotiation.cc:407] Connection header received
0929 17:50:20.166602 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.166606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.166668 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.166761 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.167945 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.168491 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.169667 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.169903 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.169975 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.172598 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.172603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.172605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.172630 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.174493 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.175100 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.175102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.175103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.175172 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.175519 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.175521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.175523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.175690 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.175758 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.176412 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.176516 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:50:20.177825 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.165034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41775 (local address 127.0.0.1:58580)
0929 17:50:20.165508 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.165562 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.166391 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.166914 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.166921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.167257 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.167767 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.167812 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.168620 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.168622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.169578 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.169587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.170028 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.170030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.170217 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.170775 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.170792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.172429 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.174615 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.174621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.174638 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.175009 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.175247 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.175250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.175252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.175439 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.175802 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.175810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.176160 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.176380 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.176724 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260929 17:50:20.179215 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab.
Entry for principal kudu/127.3.183.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab.
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
I20260929 17:50:20.184896 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:38619
--webserver_interface=127.3.183.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab
--principal=kudu/127.3.183.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:41775,127.3.183.253:37045,127.3.183.252:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:20.187824 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.179705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37354)
0929 17:50:20.179895 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.179911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.179984 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.180436 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.180439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.180597 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.180686 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.180691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.181705 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.181707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.182234 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.182241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.182826 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.182828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.182912 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.183273 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.183286 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.184764 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.186424 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.186427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.186430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.186600 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.186847 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.186849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.186851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.186930 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.187334 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.187337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.187418 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.187504 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.187623 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 17:50:20.187879 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.179808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.3.183.253:37045)
0929 17:50:20.180103 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.180112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.180130 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:50:20.180164 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.180168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.180310 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.180412 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.180777 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.181607 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.182323 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.182737 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.182815 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.184885 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.184895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.184908 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.184929 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.186323 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.186692 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.186698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.186700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.186770 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.187002 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.187005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.187007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.187240 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.187354 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.187592 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.187710 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
I20260929 17:50:20.190068 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:50:20.190945 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.190610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52484)
0929 17:50:20.190761 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.190864 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 17:50:20.191150 17232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
I20260929 17:50:20.192003 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.191764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52494)
0929 17:50:20.191888 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.191941 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 17:50:20.192147 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38619: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
I20260929 17:50:20.192463 17208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } attempt: 3
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
I20260929 17:50:20.200863 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.192935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.3.183.253:37045)
0929 17:50:20.193196 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.193198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.193207 (+ 9us) server_negotiation.cc:407] Connection header received
0929 17:50:20.193233 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.193235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.193266 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.193325 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.193731 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.194688 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.195473 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.195713 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.195819 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.197941 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.197945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.197947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.197973 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.199318 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.199717 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.199722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.199723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.199838 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.200090 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.200093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.200094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.200273 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.200354 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.200610 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.200713 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260929 17:50:20.200863 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.192880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37368)
0929 17:50:20.193000 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.193013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.193091 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.193366 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.193369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.193526 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.193628 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.193634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.194807 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.194810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.195366 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.195374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.195835 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.195839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.195951 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.196403 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.196415 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.197792 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.199422 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.199425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.199427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.199631 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.199920 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.199923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.199925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.200008 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.200358 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.200361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.200461 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.200553 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.200700 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 17:50:20.201686 17208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:50:20.202358 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.202136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52502)
0929 17:50:20.202246 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.202295 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 17:50:20.203027 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.202796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52506)
0929 17:50:20.202932 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.202974 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260929 17:50:20.203154 17208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38619: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
I20260929 17:50:20.234320 17208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } attempt: 1
I20260929 17:50:20.235225 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.234914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52518)
0929 17:50:20.235073 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.235138 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 17:50:20.236020 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.235761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52528)
0929 17:50:20.235903 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.235955 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260929 17:50:20.236146 17208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38619: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
I20260929 17:50:20.241407 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } attempt: 1
I20260929 17:50:20.242152 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.241861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52538)
0929 17:50:20.242002 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.242074 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 17:50:20.242846 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.242640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52546)
0929 17:50:20.242740 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.242788 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 17:50:20.242962 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38619: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
W20260929 17:50:20.295562 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:20.295827 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:20.295881 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:20.299995 17289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:20.300050 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:20.300068 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:20.300083 17289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:20.300097 17289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:20.305276 17289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:41775,127.3.183.253:37045,127.3.183.252:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab
--principal=kudu/127.3.183.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.183.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:20.306214 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:20.307147 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:50:20.311106 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } attempt: 2
I20260929 17:50:20.311301 17208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } attempt: 2
I20260929 17:50:20.312114 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.311673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52554)
0929 17:50:20.311954 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.312024 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 17:50:20.312383 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.311919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52556)
0929 17:50:20.312115 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.312169 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 17:50:20.312880 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.312650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52572)
0929 17:50:20.312775 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.312821 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 17:50:20.313025 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38619: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
I20260929 17:50:20.313517 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.313304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52578)
0929 17:50:20.313419 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.313461 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 17:50:20.313704 17208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38619: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38619: connect: Connection refused (error 111)
W20260929 17:50:20.316346 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:20.316349 17296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:20.316704 17289 server_base.cc:1034] running on GCE node
W20260929 17:50:20.316330 17298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:20.317333 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:20.318544 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:20.319708 17289 hybrid_clock.cc:648] HybridClock initialized: now 1790704220319675 us; error 42 us; skew 500 ppm
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:20.322494 17289 init.cc:377] Logged in from keytab as kudu/127.3.183.252@KRBTEST.COM (short username kudu)
I20260929 17:50:20.323302 17289 webserver.cc:460] Webserver started at http://127.3.183.252:38861/ using document root <none> and password file <none>
I20260929 17:50:20.323849 17289 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:20.323918 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:20.324110 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:20.326035 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data/instance:
uuid: "6528f4488ca64c0494d1d96b5c7ac9e9"
format_stamp: "Formatted at 2026-09-29 17:50:20 on dist-test-slave-wbpt"
server_key: "615f734fe558b393cfcba1e6bde2ecb9"
server_key_iv: "62daf696d2ba668e777c22bb0ed81c69"
server_key_version: "encryptionkey@0"
I20260929 17:50:20.326562 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/wal/instance:
uuid: "6528f4488ca64c0494d1d96b5c7ac9e9"
format_stamp: "Formatted at 2026-09-29 17:50:20 on dist-test-slave-wbpt"
server_key: "615f734fe558b393cfcba1e6bde2ecb9"
server_key_iv: "62daf696d2ba668e777c22bb0ed81c69"
server_key_version: "encryptionkey@0"
I20260929 17:50:20.330153 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:50:20.332463 17305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:20.333694 17289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:20.333799 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/wal
uuid: "6528f4488ca64c0494d1d96b5c7ac9e9"
format_stamp: "Formatted at 2026-09-29 17:50:20 on dist-test-slave-wbpt"
server_key: "615f734fe558b393cfcba1e6bde2ecb9"
server_key_iv: "62daf696d2ba668e777c22bb0ed81c69"
server_key_version: "encryptionkey@0"
I20260929 17:50:20.333885 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:20.374076 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:20.377137 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:20.377314 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:20.383798 17289 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.252:38619
I20260929 17:50:20.383844 17357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.252:38619 every 8 connection(s)
I20260929 17:50:20.384725 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
I20260929 17:50:20.387313 17358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:20.390877 17358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:50:20.392230 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17289
I20260929 17:50:20.392351 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704058755242-3807-0/minicluster-data/master-2/wal/instance
I20260929 17:50:20.392638 3807 external_mini_cluster.cc:1428] Setting key 4b755965cf7299b9e5e18bcc97c8c693
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:50:20.403658 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.392770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.3.183.254:41775)
0929 17:50:20.392915 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.392919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.393881 (+ 962us) server_negotiation.cc:407] Connection header received
0929 17:50:20.394651 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.394653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.394697 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.394759 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.395659 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.396199 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.397329 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.397572 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.397644 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.400213 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.400217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.400219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.400243 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.401660 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.402218 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.402221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.402223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.402301 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.402616 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.402619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.402621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.402816 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.402900 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.403358 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.403492 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:50:20.404175 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } attempt: 3
I20260929 17:50:20.404531 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.393173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41775 (local address 127.0.0.1:58588)
0929 17:50:20.393738 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.393781 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.394451 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.394828 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.394833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.395254 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.395524 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.395536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.396342 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.396346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.397217 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.397226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.397697 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.397701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.397884 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.398427 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.398443 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.400069 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.401756 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.401765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.401775 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.402133 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.402379 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.402382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.402384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.402531 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.402911 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.402917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.403158 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.403300 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.403608 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"spinlock_wait_cycles":27904,"thread_start_us":113,"threads_started":1}
I20260929 17:50:20.404546 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.393251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.3.183.252:38619)
0929 17:50:20.393622 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.393645 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.393686 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:50:20.394444 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.394466 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.394702 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.394864 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.395521 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.396594 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.397413 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.397840 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.397928 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.400607 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.400613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.400623 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.400642 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.402153 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.402548 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.402552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.402553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.402628 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.402873 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.402876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.402878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.403145 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.403325 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.403521 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.403659 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":254,"thread_start_us":128,"threads_started":1}
I20260929 17:50:20.405913 17358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:20.412779 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.404685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.3.183.252:38619)
0929 17:50:20.405029 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.405033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.405043 (+ 10us) server_negotiation.cc:407] Connection header received
0929 17:50:20.405072 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.405075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.405111 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.405176 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.405630 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.406182 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.407164 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.407481 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.407564 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.409669 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.409672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.409675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.409696 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.411192 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.411642 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.411646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.411647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.411727 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.412045 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.412048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.412050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.412224 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.412337 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.412545 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.412634 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":95,"threads_started":1}
I20260929 17:50:20.412779 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.404632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52596)
0929 17:50:20.404746 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.404759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.404843 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.405219 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.405222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.405400 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.405517 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.405524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.406277 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.406280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.407051 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.407059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.407670 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.407673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.407752 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.408229 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.408244 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.409575 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.411316 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.411320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.411322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.411552 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.411836 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.411839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.411841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.411927 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.412305 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.412308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.412401 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.412488 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.412624 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 17:50:20.412840 17208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } attempt: 3
I20260929 17:50:20.415438 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.406410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.3.183.253:37045)
0929 17:50:20.406534 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.406538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.406726 (+ 188us) server_negotiation.cc:407] Connection header received
0929 17:50:20.406816 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.406819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.406860 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.406928 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.407390 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.408036 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.409602 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.409919 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.410020 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.412252 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.412255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.412257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.412281 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.413718 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.414144 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.414147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.414149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.414219 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.414515 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.414518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.414521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.414771 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.414842 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.415206 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.415305 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:50:20.415848 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.406358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:37045 (local address 127.0.0.1:37370)
0929 17:50:20.406657 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.406669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.406741 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.407023 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.407026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.407195 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.407290 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.407295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.408164 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.408167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.409482 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.409494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.410050 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.410054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.410393 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.410775 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.410787 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.412102 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.413818 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.413821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.413822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.414049 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.414330 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.414332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.414334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.414416 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.414912 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.414916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.415021 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.415554 (+ 533us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.415675 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260929 17:50:20.416718 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: Bootstrap starting.
I20260929 17:50:20.417245 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.407057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.3.183.254:41775)
0929 17:50:20.407152 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.407155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.407544 (+ 389us) server_negotiation.cc:407] Connection header received
0929 17:50:20.407631 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.407634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.407675 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.407746 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.408396 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.408845 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.409647 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.409959 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.410019 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.414077 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.414081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.414083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.414107 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.415717 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.416178 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.416181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.416183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.416256 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.416502 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.416505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.416507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.416664 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.416853 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.416993 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.417108 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:50:20.419260 17283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:20.420215 17283 log.cc:824] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:20.420428 17358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:50:20.422407 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.413393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52602)
0929 17:50:20.413569 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.413582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.413653 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.414214 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.414217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.414378 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.414465 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.414472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.415332 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.415335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.416083 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.416092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.417072 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.417075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.417170 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.417568 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.417582 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.418574 (+ 992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.420174 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.420177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.420178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.420401 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.421187 (+ 786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.421190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.421192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.421281 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.421951 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.421954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.422057 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.422134 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.422262 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 17:50:20.422715 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: No bootstrap required, opened a new log
I20260929 17:50:20.423161 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.413859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.3.183.252:38619)
0929 17:50:20.413961 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.413965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.413978 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:50:20.414005 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.414007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.414044 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.414115 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.414599 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.415209 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.416555 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.416884 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.416969 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.418689 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.418692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.418694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.418717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.420049 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.421024 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.421028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.421029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.421107 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.421665 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.421668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.421669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.421849 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.422849 (+ 1000us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.422920 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.423027 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:50:20.426172 17208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae: Bootstrap starting.
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
I20260929 17:50:20.426676 17283 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.426846 17283 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:20.426890 17283 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7a18fe43de4823a92f88e35cf868cc, State: Initialized, Role: FOLLOWER
I20260929 17:50:20.427388 17283 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.428649 17365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.428783 17365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:50:20.428789 17283 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 17:50:20 dist-test-slave-wbpt krb5kdc[17127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704220, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:50:20.430507 17208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:20.430639 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.421323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.3.183.253:37045)
0929 17:50:20.421429 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.421433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.421444 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:50:20.421472 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.421475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.421510 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.421582 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.422226 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.422881 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.423887 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.424190 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.424257 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.426888 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.426892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.426895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.426918 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.428485 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.429051 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.429055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.429057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.429137 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.429394 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.429398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.429399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.429615 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.429829 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.430167 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.430277 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:50:20.431581 17208 log.cc:824] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:20.433318 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.420896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38619 (local address 127.0.0.1:52612)
0929 17:50:20.421061 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:20.421073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:20.421147 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:20.421453 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:20.421456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:20.421648 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:20.421722 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.421727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.422643 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.422646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.423383 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:20.423393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.424019 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.424022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:20.424095 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.428731 (+ 4636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:20.428745 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:20.430038 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:20.431947 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.431949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.431951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.432165 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.432416 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:20.432419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:20.432420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:20.432500 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:20.432887 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:20.432890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:20.432965 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 17:50:20.433054 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:50:20.433175 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 17:50:20.433933 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.420958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.3.183.252:38619)
0929 17:50:20.421043 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.421051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.421113 (+ 62us) server_negotiation.cc:407] Connection header received
0929 17:50:20.421230 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.421233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.421272 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.421348 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.421822 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.422535 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.423506 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.423838 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.423930 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.430160 (+ 6230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.430163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.430165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.430184 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.431756 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.432257 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.432259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.432261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.432338 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.432597 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.432599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.432601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.432782 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.432859 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.433674 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.433784 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 17:50:20.434242 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:50:20.433171 17215 master_runner.cc:186] Error getting master registration for 127.3.183.254:41775: OK, instance_id { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" instance_seqno: 1790704219951538 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 17:50:20.436443 17215 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:50:20.437242 17358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: Bootstrap starting.
W20260929 17:50:20.437501 17376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:50:20.437568 17376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:50:20.439306 17358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:20.440026 17358 log.cc:824] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:20.441675 17358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: No bootstrap required, opened a new log
I20260929 17:50:20.445662 17208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae: No bootstrap required, opened a new log
I20260929 17:50:20.445683 17358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.446038 17358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:20.446138 17358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6528f4488ca64c0494d1d96b5c7ac9e9, State: Initialized, Role: FOLLOWER
I20260929 17:50:20.447563 17358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.449514 17208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.449736 17208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:20.449793 17208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121dc28ce3ef40bfb306a4f7e51369ae, State: Initialized, Role: FOLLOWER
I20260929 17:50:20.450256 17208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.451560 17208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:20.453176 17358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:20.454047 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.436821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.3.183.254:41775)
0929 17:50:20.436927 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.436931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.444604 (+ 7673us) server_negotiation.cc:407] Connection header received
0929 17:50:20.444786 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.444789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.444837 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.444885 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.446426 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.447208 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.448397 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.448734 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.448809 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.450503 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.450507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.450522 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.450547 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.452288 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.452758 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.452762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.452763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.452845 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.453120 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.453123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.453126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.453324 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.453428 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.453772 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.453892 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:50:20.459798 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.436547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.3.183.253:37045)
0929 17:50:20.436648 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.436652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.444584 (+ 7932us) server_negotiation.cc:407] Connection header received
0929 17:50:20.444777 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.444780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.444830 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.444878 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.446383 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.447140 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.450506 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.451073 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.451148 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.454381 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.454385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.454388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.454415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.455957 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.457722 (+ 1765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.457725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.457727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.457809 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.458893 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.458896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.458898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.459104 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.459208 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.459505 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.459624 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 17:50:20.461097 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.462018 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:50:20.462110 17403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:50:20.462218 17403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
W20260929 17:50:20.462628 17399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:50:20.462738 17399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:50:20.463055 17403 leader_election.cc:288] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc7a18fe43de4823a92f88e35cf868cc (127.3.183.253:37045), 6528f4488ca64c0494d1d96b5c7ac9e9 (127.3.183.252:38619)
I20260929 17:50:20.463268 17387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.463397 17387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:50:20.463819 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" is_pre_election: true
I20260929 17:50:20.464107 17332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121dc28ce3ef40bfb306a4f7e51369ae in term 0.
I20260929 17:50:20.464432 17140 master_runner.cc:263] No leader master found from master 121dc28ce3ef40bfb306a4f7e51369ae
I20260929 17:50:20.464495 17140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:50:20.464749 17157 leader_election.cc:302] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121dc28ce3ef40bfb306a4f7e51369ae, 6528f4488ca64c0494d1d96b5c7ac9e9; no voters:
I20260929 17:50:20.464864 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7a18fe43de4823a92f88e35cf868cc" is_pre_election: true
I20260929 17:50:20.465041 17388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:50:20.465132 17388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:50:20.465133 17257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121dc28ce3ef40bfb306a4f7e51369ae in term 0.
I20260929 17:50:20.465185 17388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:20.465246 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:20.444214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.3.183.252:38619)
0929 17:50:20.444319 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:20.444323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:20.449464 (+ 5141us) server_negotiation.cc:407] Connection header received
0929 17:50:20.450992 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:20.450996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:20.451045 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:20.451092 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:20.454995 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.455733 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.456834 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:20.457199 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:20.457265 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:20.460206 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:20.460210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:20.460213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:20.460238 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:20.461779 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.463935 (+ 2156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.463938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.463940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.464024 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:20.464310 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:20.464313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:20.464315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:20.464537 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:20.464651 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:20.464984 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 17:50:20.465103 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:50:20.466738 17388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.467173 17388 leader_election.cc:288] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [CANDIDATE]: Term 1 election: Requested vote from peers fc7a18fe43de4823a92f88e35cf868cc (127.3.183.253:37045), 6528f4488ca64c0494d1d96b5c7ac9e9 (127.3.183.252:38619)
I20260929 17:50:20.467420 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9"
I20260929 17:50:20.467581 17332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:20.468233 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7a18fe43de4823a92f88e35cf868cc"
I20260929 17:50:20.468344 17257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:20.468864 17332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121dc28ce3ef40bfb306a4f7e51369ae in term 1.
I20260929 17:50:20.469151 17157 leader_election.cc:302] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121dc28ce3ef40bfb306a4f7e51369ae, 6528f4488ca64c0494d1d96b5c7ac9e9; no voters:
I20260929 17:50:20.469316 17388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:20.469499 17388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [term 1 LEADER]: Becoming Leader. State: Replica: 121dc28ce3ef40bfb306a4f7e51369ae, State: Running, Role: LEADER
I20260929 17:50:20.469533 17257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121dc28ce3ef40bfb306a4f7e51369ae in term 1.
I20260929 17:50:20.469816 17388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } }
I20260929 17:50:20.470490 17289 master_runner.cc:263] No leader master found from master 6528f4488ca64c0494d1d96b5c7ac9e9
I20260929 17:50:20.470557 17289 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:50:20.473133 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 121dc28ce3ef40bfb306a4f7e51369ae. Latest consensus state: current_term: 1 leader_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.473255 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: This master's current role is: LEADER
W20260929 17:50:20.475644 17411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:50:20.475721 17411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:50:20.476029 17414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:20.476898 17414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:20.483393 17332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [term 1 FOLLOWER]: Refusing update from remote peer 121dc28ce3ef40bfb306a4f7e51369ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:20.483871 17257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [term 1 FOLLOWER]: Refusing update from remote peer 121dc28ce3ef40bfb306a4f7e51369ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:20.484073 17403 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:20.484459 17388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:20.487811 17365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 121dc28ce3ef40bfb306a4f7e51369ae. Latest consensus state: current_term: 1 leader_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.487814 17387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 121dc28ce3ef40bfb306a4f7e51369ae. Latest consensus state: current_term: 1 leader_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.487941 17365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:50:20.487941 17387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:50:20.495611 17388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.495754 17388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:20.496879 17387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.497014 17387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:50:20.499557 17416 mvcc.cc:205] Tried to move back new op lower bound from 7334724487093891072 to 7334724487050653696. Current Snapshot: MvccSnapshot[applied={T|T < 7334724487093891072}]
I20260929 17:50:20.500500 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.500622 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:20.501456 17365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121dc28ce3ef40bfb306a4f7e51369ae" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41775 } } peers { permanent_uuid: "fc7a18fe43de4823a92f88e35cf868cc" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 37045 } } peers { permanent_uuid: "6528f4488ca64c0494d1d96b5c7ac9e9" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38619 } } }
I20260929 17:50:20.501575 17365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:50:20.502056 17414 catalog_manager.cc:1347] Generated new cluster ID: 2bdc4a5d920b4003b464480f4f6a3983
I20260929 17:50:20.502112 17414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:50:20.502446 17418 mvcc.cc:205] Tried to move back new op lower bound from 7334724487093891072 to 7334724487050653696. Current Snapshot: MvccSnapshot[applied={T|T < 7334724487093891072}]
I20260929 17:50:20.504320 17417 mvcc.cc:205] Tried to move back new op lower bound from 7334724487093891072 to 7334724487050653696. Current Snapshot: MvccSnapshot[applied={T|T < 7334724487093891072}]
I20260929 17:50:20.532508 17414 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:50:20.534032 17414 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:50:20.556941 17414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 121dc28ce3ef40bfb306a4f7e51369ae: Generated new TSK 0
I20260929 17:50:20.557956 17414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:50:21.438832 17376 catalog_manager.cc:1259] Loaded cluster ID: 2bdc4a5d920b4003b464480f4f6a3983
I20260929 17:50:21.438912 17376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: loading cluster ID for follower catalog manager: success
I20260929 17:50:21.440656 17376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: acquiring CA information for follower catalog manager: success
I20260929 17:50:21.441851 17376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc7a18fe43de4823a92f88e35cf868cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:50:21.477073 17411 catalog_manager.cc:1259] Loaded cluster ID: 2bdc4a5d920b4003b464480f4f6a3983
I20260929 17:50:21.477185 17411 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: loading cluster ID for follower catalog manager: success
I20260929 17:50:21.478868 17411 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: acquiring CA information for follower catalog manager: success
I20260929 17:50:21.480134 17411 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6528f4488ca64c0494d1d96b5c7ac9e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:50:21.630517 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17140
I20260929 17:50:21.636835 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17215
I20260929 17:50:21.641949 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17289
2026-09-29T17:50:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2959 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:50:21 dist-test-slave-wbpt krb5kdc[17432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:50:21 dist-test-slave-wbpt krb5kdc[17432](info): set up 2 sockets
Sep 29 17:50:21 dist-test-slave-wbpt krb5kdc[17432](info): commencing operation
krb5kdc: starting...
W20260929 17:50:22.599735 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:50:22 dist-test-slave-wbpt krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:50:22.714782 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35623
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:22.825212 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:22.825469 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:22.825510 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:22.829447 17445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:22.829511 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:22.829530 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:22.829545 17445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:22.829558 17445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:22.834810 17445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:22.835894 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:22.836917 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:22.845484 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:22.845484 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:22.845534 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:22.845775 17445 server_base.cc:1034] running on GCE node
I20260929 17:50:22.846400 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:22.847522 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:22.848690 17445 hybrid_clock.cc:648] HybridClock initialized: now 1790704222848670 us; error 34 us; skew 500 ppm
Sep 29 17:50:22 dist-test-slave-wbpt krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704222, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:22.851349 17445 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:22.852305 17445 webserver.cc:460] Webserver started at http://127.3.183.254:36555/ using document root <none> and password file <none>
I20260929 17:50:22.852849 17445 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:22.852914 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:22.853116 17445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:22.855160 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "f66f5e4f65074c8196d33d8824bf118e"
format_stamp: "Formatted at 2026-09-29 17:50:22 on dist-test-slave-wbpt"
server_key: "b1240a49036f5787cee7cc51e923946c"
server_key_iv: "979915ba3aa61d0398634f0e195b4810"
server_key_version: "encryptionkey@0"
I20260929 17:50:22.855703 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "f66f5e4f65074c8196d33d8824bf118e"
format_stamp: "Formatted at 2026-09-29 17:50:22 on dist-test-slave-wbpt"
server_key: "b1240a49036f5787cee7cc51e923946c"
server_key_iv: "979915ba3aa61d0398634f0e195b4810"
server_key_version: "encryptionkey@0"
I20260929 17:50:22.859565 17445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:50:22.862156 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:22.863479 17445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:50:22.863633 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "f66f5e4f65074c8196d33d8824bf118e"
format_stamp: "Formatted at 2026-09-29 17:50:22 on dist-test-slave-wbpt"
server_key: "b1240a49036f5787cee7cc51e923946c"
server_key_iv: "979915ba3aa61d0398634f0e195b4810"
server_key_version: "encryptionkey@0"
I20260929 17:50:22.863739 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:22.881549 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:22.884828 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:22.885013 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:22.891925 17445 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:35623
I20260929 17:50:22.891952 17512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:35623 every 8 connection(s)
I20260929 17:50:22.892886 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:22.895745 17513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:22.901016 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17445
I20260929 17:50:22.901156 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:50:22.901499 3807 external_mini_cluster.cc:1428] Setting key 9b0e206329457dade4cde67bc309be46
I20260929 17:50:22.901906 17513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e: Bootstrap starting.
I20260929 17:50:22.904616 17513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:22.905397 17513 log.cc:824] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:22.907482 17513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e: No bootstrap required, opened a new log
Sep 29 17:50:22 dist-test-slave-wbpt krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:22.910943 17513 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f5e4f65074c8196d33d8824bf118e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35623 } }
I20260929 17:50:22.911221 17513 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:22.911283 17513 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66f5e4f65074c8196d33d8824bf118e, State: Initialized, Role: FOLLOWER
I20260929 17:50:22.911718 17513 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f5e4f65074c8196d33d8824bf118e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35623 } }
I20260929 17:50:22.911885 17513 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:50:22.911952 17513 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:50:22.912048 17513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:22.912905 17513 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f5e4f65074c8196d33d8824bf118e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35623 } }
I20260929 17:50:22.913213 17513 leader_election.cc:302] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66f5e4f65074c8196d33d8824bf118e; no voters:
I20260929 17:50:22.913524 17513 leader_election.cc:288] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:50:22.913607 17518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:22.913771 17518 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [term 1 LEADER]: Becoming Leader. State: Replica: f66f5e4f65074c8196d33d8824bf118e, State: Running, Role: LEADER
I20260929 17:50:22.914132 17518 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f5e4f65074c8196d33d8824bf118e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35623 } }
I20260929 17:50:22.914816 17513 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:22.915659 17519 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66f5e4f65074c8196d33d8824bf118e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f5e4f65074c8196d33d8824bf118e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35623 } } }
I20260929 17:50:22.915933 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66f5e4f65074c8196d33d8824bf118e. Latest consensus state: current_term: 1 leader_uuid: "f66f5e4f65074c8196d33d8824bf118e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f5e4f65074c8196d33d8824bf118e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 35623 } } }
I20260929 17:50:22.916009 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:22.916055 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:22.902973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.3.183.254:35623)
0929 17:50:22.903447 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:22.903463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:22.903501 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:50:22.904282 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:22.904304 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:22.904663 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:22.905005 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:22.905874 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:22.906858 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:22.907724 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:22.908243 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:22.908391 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:22.910694 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:22.910699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:22.910712 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:22.910739 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:22.912574 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:22.912993 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:22.912999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:22.913005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:22.913151 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:22.913409 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:22.913413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:22.913415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:22.913914 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:22.914651 (+ 737us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:22.914778 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 17:50:22.915044 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":165,"threads_started":1}
I20260929 17:50:22.916504 17519 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:22.916971 17527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:22.919108 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:50:22.920181 17527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:22.925937 17527 catalog_manager.cc:1347] Generated new cluster ID: 7bbd72c6a3494141a9800cef6cf51d20
I20260929 17:50:22.926029 17527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:50:22.951454 17527 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:50:22.952869 17527 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:50:22.964140 17527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66f5e4f65074c8196d33d8824bf118e: Generated new TSK 0
I20260929 17:50:22.964900 17527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:50:22.974632 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17445
2026-09-29T17:50:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1331 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:50:23 dist-test-slave-wbpt krb5kdc[17540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:50:23 dist-test-slave-wbpt krb5kdc[17540](info): set up 2 sockets
Sep 29 17:50:23 dist-test-slave-wbpt krb5kdc[17540](info): commencing operation
krb5kdc: starting...
W20260929 17:50:23.931708 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:50:24.056363 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46883
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:24.167119 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:24.167408 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:24.167449 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:24.171471 17553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:24.171535 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:24.171553 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:24.171569 17553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:24.171582 17553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:24.177028 17553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:24.178133 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:24.179155 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:24.187155 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:24.187186 17559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:24.187192 17561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:24.187325 17553 server_base.cc:1034] running on GCE node
I20260929 17:50:24.187940 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:24.189029 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:24.190174 17553 hybrid_clock.cc:648] HybridClock initialized: now 1790704224190153 us; error 35 us; skew 500 ppm
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:24.192808 17553 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:24.193660 17553 webserver.cc:460] Webserver started at http://127.3.183.254:42413/ using document root <none> and password file <none>
I20260929 17:50:24.194175 17553 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:24.194238 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:24.194424 17553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:24.196491 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d35b618f759d2bcc80905aab8bf63092"
server_key_iv: "c62340d74ba2d53fb86d65c5f52bf2dc"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.197010 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d35b618f759d2bcc80905aab8bf63092"
server_key_iv: "c62340d74ba2d53fb86d65c5f52bf2dc"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.200592 17553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:50:24.202936 17568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.204210 17553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:24.204324 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d35b618f759d2bcc80905aab8bf63092"
server_key_iv: "c62340d74ba2d53fb86d65c5f52bf2dc"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.204417 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:24.217584 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:24.220566 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:24.220737 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:24.227294 17553 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:46883
I20260929 17:50:24.227537 17620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:46883 every 8 connection(s)
I20260929 17:50:24.228241 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:24.230840 17621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:24.232589 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17553
I20260929 17:50:24.232718 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:50:24.233062 3807 external_mini_cluster.cc:1428] Setting key f9714ba55fb701e6aaba7081a1dc1ab8
I20260929 17:50:24.237210 17621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Bootstrap starting.
I20260929 17:50:24.239344 17621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:24.240078 17621 log.cc:824] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:24.242166 17621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: No bootstrap required, opened a new log
I20260929 17:50:24.246572 17621 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:24.246851 17621 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:24.246905 17621 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc89abc8edf94fa4ad37c3f4e327a8c4, State: Initialized, Role: FOLLOWER
I20260929 17:50:24.247376 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.234502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.3.183.254:46883)
0929 17:50:24.234972 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.234989 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.235025 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:50:24.235664 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.235681 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.236065 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.236360 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:24.237133 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.238131 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.238967 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.239451 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.239593 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.242328 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:24.242334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:24.242351 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:24.242381 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:24.244325 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.244762 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.244768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.244774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.244909 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.245129 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.245132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.245134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.245642 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:24.245847 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:24.246108 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 17:50:24.246408 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260929 17:50:24.247390 17621 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:24.247552 17621 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:50:24.247620 17621 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:50:24.247713 17621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:24.248908 17621 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:24.249246 17621 leader_election.cc:302] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc89abc8edf94fa4ad37c3f4e327a8c4; no voters:
I20260929 17:50:24.249547 17621 leader_election.cc:288] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:50:24.249648 17626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:24.249858 17626 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 LEADER]: Becoming Leader. State: Replica: fc89abc8edf94fa4ad37c3f4e327a8c4, State: Running, Role: LEADER
I20260929 17:50:24.250191 17626 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:24.250710 17621 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:24.251291 17626 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc89abc8edf94fa4ad37c3f4e327a8c4. Latest consensus state: current_term: 1 leader_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } } }
I20260929 17:50:24.251399 17626 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:24.251686 17627 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } } }
I20260929 17:50:24.251817 17632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:24.251830 17627 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:24.254666 17632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:24.260416 17632 catalog_manager.cc:1347] Generated new cluster ID: 9a29fbd0c1cd4659a5c41a8c43aa5665
I20260929 17:50:24.260496 17632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:50:24.283043 17632 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:50:24.284422 17632 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:50:24.290297 17632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Generated new TSK 0
I20260929 17:50:24.291093 17632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:50:24.323666 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46883
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:24.430104 17647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:24.430358 17647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:24.430402 17647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:24.434371 17647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:24.434434 17647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:24.434504 17647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:50:24.440030 17647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:24.441062 17647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:24.442070 17647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:24.451586 17652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:24.451635 17655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:24.451647 17647 server_base.cc:1034] running on GCE node
W20260929 17:50:24.451874 17653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:24.452245 17647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:24.452831 17647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:24.453994 17647 hybrid_clock.cc:648] HybridClock initialized: now 1790704224453976 us; error 61 us; skew 500 ppm
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:24.456539 17647 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:50:24.457402 17647 webserver.cc:460] Webserver started at http://127.3.183.193:45821/ using document root <none> and password file <none>
I20260929 17:50:24.457917 17647 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:24.457995 17647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:24.458199 17647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:24.460240 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d2497be67de89701e86d0cf70b588fd3"
server_key_iv: "90479cb556a18a52a539b647fab4e903"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.460760 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d2497be67de89701e86d0cf70b588fd3"
server_key_iv: "90479cb556a18a52a539b647fab4e903"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.464241 17647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:50:24.466538 17662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.467705 17647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:24.467849 17647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d2497be67de89701e86d0cf70b588fd3"
server_key_iv: "90479cb556a18a52a539b647fab4e903"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.467940 17647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:24.486513 17647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:24.489606 17647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:24.489777 17647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:24.490350 17647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:24.491143 17647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:24.491199 17647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.491252 17647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:24.491278 17647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.499920 17647 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:42523
I20260929 17:50:24.499956 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:42523 every 8 connection(s)
I20260929 17:50:24.500772 17647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:24.509856 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17647
I20260929 17:50:24.509991 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:50:24.510314 3807 external_mini_cluster.cc:1428] Setting key f86351cc57c2bd2bc24726dd2172a5f9
I20260929 17:50:24.513759 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.502420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:39161 (local address 127.3.183.254:46883)
0929 17:50:24.502551 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.502555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.503319 (+ 764us) server_negotiation.cc:407] Connection header received
0929 17:50:24.504595 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.504598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.504656 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.504725 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:24.505872 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.506408 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.507623 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.507898 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.507990 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.510401 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:24.510405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:24.510408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:24.510436 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:24.511900 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.512359 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.512362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.512364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.512442 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.512751 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.512754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.512756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.512932 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:24.513019 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:24.513487 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 17:50:24.513606 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:50:24.514549 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.502777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:39161)
0929 17:50:24.503181 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:24.503223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:24.504388 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:24.504897 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:24.504905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:24.505256 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:24.505731 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.505745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.506529 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.506532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.507514 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.507524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.508035 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.508039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.508205 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.508724 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:24.508739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:24.510235 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:24.512004 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:24.512010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:24.512025 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:24.512265 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:24.512528 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:24.512531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:24.512533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:24.512665 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:24.513028 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:24.513034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:24.513261 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 17:50:24.513422 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 17:50:24.513734 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260929 17:50:24.515831 17776 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46883
I20260929 17:50:24.516093 17776 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:24.516675 17776 heartbeater.cc:511] Master 127.3.183.254:46883 requested a full tablet report, sending...
I20260929 17:50:24.518195 17585 ts_manager.cc:194] Registered new tserver with Master: 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523)
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:50:24.519322 17585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:39161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:50:24.535153 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46883
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:24.643675 17781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:24.643985 17781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:24.644030 17781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:24.647992 17781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:24.648046 17781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:24.648113 17781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:50:24.653430 17781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:24.654466 17781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:24.655452 17781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:24.665009 17789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:24.665009 17786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:24.665040 17781 server_base.cc:1034] running on GCE node
W20260929 17:50:24.665009 17787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:24.665642 17781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:24.666277 17781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:24.667445 17781 hybrid_clock.cc:648] HybridClock initialized: now 1790704224667426 us; error 33 us; skew 500 ppm
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:24.670145 17781 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:50:24.671087 17781 webserver.cc:460] Webserver started at http://127.3.183.194:43977/ using document root <none> and password file <none>
I20260929 17:50:24.671664 17781 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:24.671737 17781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:24.671967 17781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:24.674010 17781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "a5583a1f83704ef7b2290a51c639c374"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "08833794a352528c877c7be7b96d92e3"
server_key_iv: "f922e1291201861b52b1aa789f62b3ed"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.674564 17781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "a5583a1f83704ef7b2290a51c639c374"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "08833794a352528c877c7be7b96d92e3"
server_key_iv: "f922e1291201861b52b1aa789f62b3ed"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.678300 17781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 17:50:24.680685 17796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.681875 17781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:24.681988 17781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "a5583a1f83704ef7b2290a51c639c374"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "08833794a352528c877c7be7b96d92e3"
server_key_iv: "f922e1291201861b52b1aa789f62b3ed"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.682086 17781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:24.718966 17781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:24.722237 17781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:24.722427 17781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:24.723049 17781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:24.723894 17781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:24.723941 17781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.723994 17781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:24.724023 17781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.732539 17781 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:42193
I20260929 17:50:24.732563 17909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:42193 every 8 connection(s)
I20260929 17:50:24.733434 17781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:50:24.741742 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17781
I20260929 17:50:24.741863 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:50:24.742139 3807 external_mini_cluster.cc:1428] Setting key 22a91dbe897878a6ad5651cd9347b8c9
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:24.747071 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.735253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:37509 (local address 127.3.183.254:46883)
0929 17:50:24.735395 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.735409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.736323 (+ 914us) server_negotiation.cc:407] Connection header received
0929 17:50:24.737536 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.737539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.737587 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.737664 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:24.738825 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.739382 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.740616 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.740864 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.740978 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.743553 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:24.743556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:24.743558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:24.743582 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:24.745023 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.745561 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.745564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.745565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.745645 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.745977 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.745981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.745983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.746172 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:24.746258 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:24.746772 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 17:50:24.746905 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:50:24.748029 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.735687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:37509)
0929 17:50:24.736173 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:24.736216 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:24.737335 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:24.737798 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:24.737805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:24.738167 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:24.738662 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.738676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.739531 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.739534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.740477 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.740486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.741003 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.741005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.741149 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.741716 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:24.741732 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:24.743374 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:24.745140 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:24.745145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:24.745159 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:24.745459 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:24.745733 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:24.745736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:24.745738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:24.745879 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:24.746271 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:24.746278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:24.746535 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 17:50:24.746710 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 17:50:24.747070 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":110,"threads_started":1}
I20260929 17:50:24.749208 17910 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46883
I20260929 17:50:24.749439 17910 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:24.749949 17910 heartbeater.cc:511] Master 127.3.183.254:46883 requested a full tablet report, sending...
I20260929 17:50:24.751022 17585 ts_manager.cc:194] Registered new tserver with Master: a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193)
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:50:24.751688 17585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:37509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:50:24.771036 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46883
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:24.881982 17915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:24.882261 17915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:24.882303 17915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:24.886394 17915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:24.886456 17915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:24.886515 17915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:50:24.891819 17915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:24.892877 17915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:24.893898 17915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:24.903867 17921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:24.904006 17915 server_base.cc:1034] running on GCE node
W20260929 17:50:24.903848 17920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:24.903936 17923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:24.904531 17915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:24.905201 17915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:24.906388 17915 hybrid_clock.cc:648] HybridClock initialized: now 1790704224906357 us; error 39 us; skew 500 ppm
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:24.909317 17915 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:50:24.910174 17915 webserver.cc:460] Webserver started at http://127.3.183.195:46813/ using document root <none> and password file <none>
I20260929 17:50:24.910691 17915 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:24.910748 17915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:24.910909 17915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:24.912947 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "db73a96a78d94374b3b5aacff24aab9b"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "4fb92ccde4b74c3c3be89781cdf3c985"
server_key_iv: "2e31344f29ee75e6e766336b71238f67"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.913450 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "db73a96a78d94374b3b5aacff24aab9b"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "4fb92ccde4b74c3c3be89781cdf3c985"
server_key_iv: "2e31344f29ee75e6e766336b71238f67"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.917033 17915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:50:24.919335 17930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.920562 17915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:50:24.920656 17915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "db73a96a78d94374b3b5aacff24aab9b"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "4fb92ccde4b74c3c3be89781cdf3c985"
server_key_iv: "2e31344f29ee75e6e766336b71238f67"
server_key_version: "encryptionkey@0"
I20260929 17:50:24.920737 17915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:24.933612 17915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:24.936688 17915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:24.936838 17915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:24.937426 17915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:24.938208 17915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:24.938252 17915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.938295 17915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:24.938323 17915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:24.946319 17915 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:39517
I20260929 17:50:24.946359 18043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:39517 every 8 connection(s)
I20260929 17:50:24.947176 17915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:50:24 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704224, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:24.957382 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 17915
I20260929 17:50:24.957511 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:50:24.957808 3807 external_mini_cluster.cc:1428] Setting key 659306e7ce9d661611c2bdabe7d9e3af
I20260929 17:50:24.960758 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.948931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:59667 (local address 127.3.183.254:46883)
0929 17:50:24.949093 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.949096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.949873 (+ 777us) server_negotiation.cc:407] Connection header received
0929 17:50:24.951180 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.951184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.951239 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.951311 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:24.952651 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.953331 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.954453 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.954665 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.954752 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.957057 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:24.957060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:24.957062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:24.957085 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:24.958558 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.959135 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.959138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.959140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.959226 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.959630 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.959633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.959634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.959855 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:24.959946 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:24.960536 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 17:50:24.960642 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 17:50:24.961499 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.949303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:59667)
0929 17:50:24.949734 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:24.949775 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:24.950986 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:24.951493 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:24.951499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:24.951905 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:24.952506 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.952525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.953448 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.953451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.954324 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.954333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.954800 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.954803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.954981 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.955461 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:24.955477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:24.956904 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:24.958679 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:24.958684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:24.958700 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:24.959025 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:24.959324 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:24.959327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:24.959329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:24.959539 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:24.960002 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:24.960008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:24.960276 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 17:50:24.960447 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 17:50:24.960710 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260929 17:50:24.962476 18044 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46883
I20260929 17:50:24.962699 18044 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:24.963169 18044 heartbeater.cc:511] Master 127.3.183.254:46883 requested a full tablet report, sending...
I20260929 17:50:24.964295 17585 ts_manager.cc:194] Registered new tserver with Master: db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517)
I20260929 17:50:24.964835 17585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:59667
I20260929 17:50:24.971841 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:50:24.980569 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.973740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.3.183.254:46883)
0929 17:50:24.973870 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.973875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.974052 (+ 177us) server_negotiation.cc:407] Connection header received
0929 17:50:24.974158 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.974160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.974199 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.974257 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:24.974912 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.975412 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.976256 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.976488 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.976564 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.977935 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:24.977938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:24.977940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:24.977962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:24.979094 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.979468 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.979471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.979472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.979549 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:24.979737 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:24.979739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:24.979740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:24.979982 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:24.980082 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:24.980297 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 17:50:24.980413 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:50:24.983273 17585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:50:24.986431 17585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:50:24.999998 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.994546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42523 (local address 127.0.0.1:40000)
0929 17:50:24.994992 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:24.995040 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:24.995180 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:24.996074 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:24.996080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:24.996103 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:24.996255 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.996262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.997506 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.997510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.998670 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.998680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.999445 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.999449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.999554 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.999595 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 17:50:24.999701 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:50:24.999837 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260929 17:50:24.999993 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.994690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.3.183.193:42523)
0929 17:50:24.994934 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.994951 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.995137 (+ 186us) server_negotiation.cc:407] Connection header received
0929 17:50:24.995645 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.995652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.995900 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.996396 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:24.996542 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.997385 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.998782 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.999328 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.999450 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:24.999604 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:24.999725 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 17:50:24.999842 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260929 17:50:25.000751 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.995529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39517 (local address 127.0.0.1:42230)
0929 17:50:24.995853 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:24.995888 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:24.995964 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:24.996569 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:24.996572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:24.996590 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:24.996674 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.996679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.997985 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.997988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.999256 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.999267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.000377 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.000381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.000478 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.000502 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.000555 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.000629 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":36,"thread_start_us":82,"threads_started":1}
I20260929 17:50:25.000905 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.995537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.3.183.194:42193)
0929 17:50:24.995945 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.995958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.995978 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:50:24.996048 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.996053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.996236 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.996388 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:24.996636 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.997792 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.999383 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.000297 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.000427 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.000576 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.000671 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.000759 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":102,"threads_started":1}
I20260929 17:50:25.001564 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.995931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.3.183.195:39517)
0929 17:50:24.996191 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:24.996202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:24.996219 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:50:24.996282 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:24.996285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:24.996448 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:24.996598 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:24.996767 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.997852 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.999393 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.000244 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.001251 (+ 1007us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.001353 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.001392 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.001459 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
I20260929 17:50:25.001655 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:24.995199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42193 (local address 127.0.0.1:45388)
0929 17:50:24.995567 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:24.995579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:24.995662 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:24.996377 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:24.996380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:24.996398 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:24.996516 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.996523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:24.997944 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:24.997948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:24.999259 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:24.999267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.000432 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.000436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.000540 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.000563 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.000643 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.000714 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":92,"thread_start_us":97,"threads_started":1}
I20260929 17:50:25.003258 17710 tablet_service.cc:1467] Processing CreateTablet for tablet 74cbf018a64a4789816f56d3fb8ccfb1 (DEFAULT_TABLE table=test-table [id=82e932dc10e64744925665c2ea482b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:25.004240 17710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cbf018a64a4789816f56d3fb8ccfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:25.004199 17978 tablet_service.cc:1467] Processing CreateTablet for tablet 74cbf018a64a4789816f56d3fb8ccfb1 (DEFAULT_TABLE table=test-table [id=82e932dc10e64744925665c2ea482b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:25.004199 17844 tablet_service.cc:1467] Processing CreateTablet for tablet 74cbf018a64a4789816f56d3fb8ccfb1 (DEFAULT_TABLE table=test-table [id=82e932dc10e64744925665c2ea482b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:25.005204 17844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cbf018a64a4789816f56d3fb8ccfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:25.005210 17978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cbf018a64a4789816f56d3fb8ccfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:25.010496 18061 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap starting.
I20260929 17:50:25.012071 18063 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Bootstrap starting.
I20260929 17:50:25.012485 18062 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Bootstrap starting.
I20260929 17:50:25.012909 18061 tablet_bootstrap.cc:654] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:25.014144 18061 log.cc:824] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:25.014951 18062 tablet_bootstrap.cc:654] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:25.016328 18061 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: No bootstrap required, opened a new log
I20260929 17:50:25.016453 18062 log.cc:824] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:25.016531 18061 ts_tablet_manager.cc:1397] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:50:25.017110 18063 tablet_bootstrap.cc:654] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:25.018065 18062 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: No bootstrap required, opened a new log
I20260929 17:50:25.018239 18062 ts_tablet_manager.cc:1397] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:50:25.018226 18063 log.cc:824] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:25.020251 18063 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: No bootstrap required, opened a new log
I20260929 17:50:25.020445 18063 ts_tablet_manager.cc:1397] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 17:50:25.020318 18061 raft_consensus.cc:348] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.020603 18061 raft_consensus.cc:374] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:25.020700 18061 raft_consensus.cc:729] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fcbfb2e9fc043dea44ceaf18cfe4f93, State: Initialized, Role: FOLLOWER
I20260929 17:50:25.021158 18061 consensus_queue.cc:260] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.021306 18062 raft_consensus.cc:348] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.021607 18062 raft_consensus.cc:374] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:25.021688 18062 raft_consensus.cc:729] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5583a1f83704ef7b2290a51c639c374, State: Initialized, Role: FOLLOWER
I20260929 17:50:25.021835 17776 heartbeater.cc:503] Master 127.3.183.254:46883 was elected leader, sending a full tablet report...
I20260929 17:50:25.022145 18061 ts_tablet_manager.cc:1428] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:50:25.022176 18062 consensus_queue.cc:260] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.022811 17910 heartbeater.cc:503] Master 127.3.183.254:46883 was elected leader, sending a full tablet report...
I20260929 17:50:25.023105 18062 ts_tablet_manager.cc:1428] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:50:25.025259 18063 raft_consensus.cc:348] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.025560 18063 raft_consensus.cc:374] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:25.025658 18063 raft_consensus.cc:729] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db73a96a78d94374b3b5aacff24aab9b, State: Initialized, Role: FOLLOWER
I20260929 17:50:25.026089 18063 consensus_queue.cc:260] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.026976 18063 ts_tablet_manager.cc:1428] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:50:25.027307 18044 heartbeater.cc:503] Master 127.3.183.254:46883 was elected leader, sending a full tablet report...
I20260929 17:50:25.110673 18067 raft_consensus.cc:482] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:50:25.110874 18067 raft_consensus.cc:504] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.112043 18067 leader_election.cc:288] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193), db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517)
I20260929 17:50:25.116616 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.112279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42193 (local address 127.3.183.193:47105)
0929 17:50:25.112555 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.112569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:25.112739 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:25.113138 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:25.113141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:25.113160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:25.113302 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.113310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.114143 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.114145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.115511 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.115522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.116208 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.116212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.116307 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.116331 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.116395 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.116454 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":31,"thread_start_us":103,"threads_started":1}
I20260929 17:50:25.116626 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.112408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:47105 (local address 127.3.183.194:42193)
0929 17:50:25.112550 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.112556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.112651 (+ 95us) server_negotiation.cc:407] Connection header received
0929 17:50:25.112877 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.112881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.112937 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.112993 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:25.113433 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.114032 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.115636 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.116088 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.116183 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.116236 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.116423 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.116503 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:50:25.116712 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.112860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:37455 (local address 127.3.183.195:39517)
0929 17:50:25.113026 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.113030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.113154 (+ 124us) server_negotiation.cc:407] Connection header received
0929 17:50:25.113285 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.113289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.113332 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.113400 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:25.113649 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.114132 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.115673 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.116131 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.116263 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.116313 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.116534 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.116607 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:50:25.116744 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.112713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39517 (local address 127.3.183.193:37455)
0929 17:50:25.113078 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.113090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:25.113190 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:25.113434 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:25.113436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:25.113453 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:25.113553 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.113559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.114233 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.114236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.115511 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.115522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.116269 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.116273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.116367 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.116387 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.116504 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.116571 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":33,"thread_start_us":90,"threads_started":1}
I20260929 17:50:25.117420 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cbf018a64a4789816f56d3fb8ccfb1" candidate_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5583a1f83704ef7b2290a51c639c374" is_pre_election: true
I20260929 17:50:25.117429 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cbf018a64a4789816f56d3fb8ccfb1" candidate_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db73a96a78d94374b3b5aacff24aab9b" is_pre_election: true
I20260929 17:50:25.117707 17864 raft_consensus.cc:2454] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fcbfb2e9fc043dea44ceaf18cfe4f93 in term 0.
I20260929 17:50:25.117712 17998 raft_consensus.cc:2454] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fcbfb2e9fc043dea44ceaf18cfe4f93 in term 0.
I20260929 17:50:25.118125 17664 leader_election.cc:302] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fcbfb2e9fc043dea44ceaf18cfe4f93, a5583a1f83704ef7b2290a51c639c374; no voters:
I20260929 17:50:25.118366 18067 raft_consensus.cc:2781] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:50:25.118453 18067 raft_consensus.cc:482] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:50:25.118503 18067 raft_consensus.cc:3037] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:25.119570 18067 raft_consensus.cc:504] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.119910 18067 leader_election.cc:288] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 1 election: Requested vote from peers a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193), db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517)
I20260929 17:50:25.120292 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cbf018a64a4789816f56d3fb8ccfb1" candidate_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db73a96a78d94374b3b5aacff24aab9b"
I20260929 17:50:25.120283 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cbf018a64a4789816f56d3fb8ccfb1" candidate_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5583a1f83704ef7b2290a51c639c374"
I20260929 17:50:25.120407 17864 raft_consensus.cc:3037] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:25.120460 17998 raft_consensus.cc:3037] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:25.121676 17998 raft_consensus.cc:2454] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fcbfb2e9fc043dea44ceaf18cfe4f93 in term 1.
I20260929 17:50:25.121695 17864 raft_consensus.cc:2454] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fcbfb2e9fc043dea44ceaf18cfe4f93 in term 1.
I20260929 17:50:25.121955 17664 leader_election.cc:302] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fcbfb2e9fc043dea44ceaf18cfe4f93, a5583a1f83704ef7b2290a51c639c374; no voters:
I20260929 17:50:25.122118 18067 raft_consensus.cc:2781] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:25.122319 18067 raft_consensus.cc:686] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 LEADER]: Becoming Leader. State: Replica: 6fcbfb2e9fc043dea44ceaf18cfe4f93, State: Running, Role: LEADER
I20260929 17:50:25.122661 18067 consensus_queue.cc:237] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.125816 17584 catalog_manager.cc:5495] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } health_report { overall_health: UNKNOWN } } }
I20260929 17:50:25.151474 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.147978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.3.183.193:42523)
0929 17:50:25.148121 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.148125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.148137 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:50:25.148233 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.148235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.148299 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.148383 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:50:25.148646 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.149168 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.149984 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.150204 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.150280 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.150589 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:50:25.151105 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:50:25.151207 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.151295 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.151349 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:50:25.157354 17998 raft_consensus.cc:1261] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 1 FOLLOWER]: Refusing update from remote peer 6fcbfb2e9fc043dea44ceaf18cfe4f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:25.157354 17864 raft_consensus.cc:1261] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Refusing update from remote peer 6fcbfb2e9fc043dea44ceaf18cfe4f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:25.158133 18072 consensus_queue.cc:1035] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:25.158326 18067 consensus_queue.cc:1035] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:25.173905 17583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:25.174095 17583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:25.176457 17583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Sending DeleteTablet for 3 replicas of tablet 74cbf018a64a4789816f56d3fb8ccfb1
I20260929 17:50:25.177181 17710 tablet_service.cc:1514] Processing DeleteTablet for tablet 74cbf018a64a4789816f56d3fb8ccfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:25 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:40000
I20260929 17:50:25.177300 17978 tablet_service.cc:1514] Processing DeleteTablet for tablet 74cbf018a64a4789816f56d3fb8ccfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:25 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42230
I20260929 17:50:25.177685 18085 tablet_replica.cc:330] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: stopping tablet replica
I20260929 17:50:25.177708 18086 tablet_replica.cc:330] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: stopping tablet replica
I20260929 17:50:25.177897 17844 tablet_service.cc:1514] Processing DeleteTablet for tablet 74cbf018a64a4789816f56d3fb8ccfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:25 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45388
I20260929 17:50:25.178009 18085 raft_consensus.cc:2229] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:50:25.178020 18086 raft_consensus.cc:2229] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:50:25.178314 18086 raft_consensus.cc:2258] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:25.178330 18085 raft_consensus.cc:2258] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:25.178606 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17647
I20260929 17:50:25.179797 18087 tablet_replica.cc:330] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: stopping tablet replica
I20260929 17:50:25.180110 18087 raft_consensus.cc:2229] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:50:25.180398 18087 raft_consensus.cc:2258] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:25.181577 18087 ts_tablet_manager.cc:1905] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:50:25.181586 18086 ts_tablet_manager.cc:1905] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:50:25.184230 18087 ts_tablet_manager.cc:1918] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:50:25.184230 18086 ts_tablet_manager.cc:1918] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:50:25.184329 18086 log.cc:1196] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/74cbf018a64a4789816f56d3fb8ccfb1
I20260929 17:50:25.184437 18087 log.cc:1196] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/74cbf018a64a4789816f56d3fb8ccfb1
I20260929 17:50:25.184674 18086 ts_tablet_manager.cc:1939] T 74cbf018a64a4789816f56d3fb8ccfb1 P db73a96a78d94374b3b5aacff24aab9b: Deleting consensus metadata
W20260929 17:50:25.184717 17570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:42523 (error 108)
I20260929 17:50:25.184814 18087 ts_tablet_manager.cc:1939] T 74cbf018a64a4789816f56d3fb8ccfb1 P a5583a1f83704ef7b2290a51c639c374: Deleting consensus metadata
I20260929 17:50:25.185707 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17781
I20260929 17:50:25.185730 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.185472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42523 (local address 127.0.0.1:40016)
0929 17:50:25.185575 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.185644 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 17:50:25.186807 17570 catalog_manager.cc:4584] TS 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74cbf018a64a4789816f56d3fb8ccfb1: Network error: Client connection negotiation failed: client connection to 127.3.183.193:42523: connect: Connection refused (error 111)
I20260929 17:50:25.190954 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17915
I20260929 17:50:25.190958 17570 catalog_manager.cc:4841] TS db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517): tablet 74cbf018a64a4789816f56d3fb8ccfb1 (table test-table [id=82e932dc10e64744925665c2ea482b06]) successfully deleted
I20260929 17:50:25.191625 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.191237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42193 (local address 127.0.0.1:45396)
0929 17:50:25.191485 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.191543 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 17:50:25.195844 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 17553
I20260929 17:50:25.201268 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46883
--webserver_interface=127.3.183.254
--webserver_port=42413
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:25.308266 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:25.308594 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:25.308668 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:25.313271 18089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:25.313395 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:25.313438 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:25.313464 18089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:25.313485 18089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:25.322073 18089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=42413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:25.323993 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:25.325556 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:25.339931 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:25.339936 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:25.339931 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:25.340277 18089 server_base.cc:1034] running on GCE node
I20260929 17:50:25.340864 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:25.342074 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:25.343258 18089 hybrid_clock.cc:648] HybridClock initialized: now 1790704225343238 us; error 36 us; skew 500 ppm
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:25.346041 18089 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:25.346990 18089 webserver.cc:460] Webserver started at http://127.3.183.254:42413/ using document root <none> and password file <none>
I20260929 17:50:25.347537 18089 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:25.347620 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:25.351548 18089 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:50:25.353662 18105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.354975 18089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:50:25.355105 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d35b618f759d2bcc80905aab8bf63092"
server_key_iv: "c62340d74ba2d53fb86d65c5f52bf2dc"
server_key_version: "encryptionkey@0"
I20260929 17:50:25.355715 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:25.371513 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:25.374517 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:25.374699 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:25.381587 18089 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:46883
I20260929 17:50:25.381857 18157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:46883 every 8 connection(s)
I20260929 17:50:25.382635 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:25.386808 18158 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:50:25.387223 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18089
I20260929 17:50:25.387610 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:42523
--local_ip_for_outbound_sockets=127.3.183.193
--tserver_master_addrs=127.3.183.254:46883
--webserver_port=45821
--webserver_interface=127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:25.388463 18158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Bootstrap starting.
I20260929 17:50:25.405323 18158 log.cc:824] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:25.413915 18158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:50:25.414260 18158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Bootstrap complete.
I20260929 17:50:25.418139 18158 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:25.418540 18158 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc89abc8edf94fa4ad37c3f4e327a8c4, State: Initialized, Role: FOLLOWER
I20260929 17:50:25.418990 18158 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:25.419116 18158 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:50:25.419184 18158 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:50:25.419276 18158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:50:25.421137 18158 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:25.421439 18158 leader_election.cc:302] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc89abc8edf94fa4ad37c3f4e327a8c4; no voters:
I20260929 17:50:25.421721 18158 leader_election.cc:288] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:50:25.421818 18162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:50:25.422011 18162 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 LEADER]: Becoming Leader. State: Replica: fc89abc8edf94fa4ad37c3f4e327a8c4, State: Running, Role: LEADER
I20260929 17:50:25.422355 18162 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:25.422961 18158 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:25.423290 18162 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc89abc8edf94fa4ad37c3f4e327a8c4. Latest consensus state: current_term: 2 leader_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } } }
I20260929 17:50:25.423485 18162 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:25.423663 18163 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } } }
I20260929 17:50:25.423871 18163 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:25.423936 18166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:25.428591 18166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:25.429371 18166 catalog_manager.cc:1259] Loaded cluster ID: 9a29fbd0c1cd4659a5c41a8c43aa5665
I20260929 17:50:25.429438 18166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:50:25.430781 18166 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 17:50:25.435525 18166 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 17:50:25.440786 18166 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:50:25.441504 18166 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Loaded TSK: 0
I20260929 17:50:25.442682 18166 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:50:25.498562 18160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:25.498816 18160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:25.498872 18160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:25.502849 18160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:25.502907 18160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:25.502985 18160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:50:25.508102 18160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:42523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=45821
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:25.509150 18160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:25.510166 18160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:25.519390 18185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:25.519390 18186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:25.519493 18188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:25.519865 18160 server_base.cc:1034] running on GCE node
I20260929 17:50:25.520320 18160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:25.520912 18160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:25.522071 18160 hybrid_clock.cc:648] HybridClock initialized: now 1790704225522045 us; error 47 us; skew 500 ppm
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:25.524869 18160 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:50:25.525816 18160 webserver.cc:460] Webserver started at http://127.3.183.193:45821/ using document root <none> and password file <none>
I20260929 17:50:25.526348 18160 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:25.526420 18160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:25.530529 18160 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:50:25.532958 18195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.534502 18160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:50:25.534636 18160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d2497be67de89701e86d0cf70b588fd3"
server_key_iv: "90479cb556a18a52a539b647fab4e903"
server_key_version: "encryptionkey@0"
I20260929 17:50:25.535199 18160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:25.561532 18160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:25.564700 18160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:25.564877 18160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:25.565470 18160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:25.566581 18202 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:50:25.570574 18160 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:50:25.570645 18160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 17:50:25.570712 18160 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:50:25.572518 18160 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:50:25.572566 18160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 17:50:25.572839 18202 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap starting.
I20260929 17:50:25.581629 18160 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:42523
I20260929 17:50:25.581707 18309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:42523 every 8 connection(s)
I20260929 17:50:25.582599 18160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:50:25.584141 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18160
I20260929 17:50:25.584663 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:42193
--local_ip_for_outbound_sockets=127.3.183.194
--tserver_master_addrs=127.3.183.254:46883
--webserver_port=43977
--webserver_interface=127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:25.591725 18202 log.cc:824] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:25.599402 18202 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:50:25.599826 18202 tablet_bootstrap.cc:492] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap complete.
I20260929 17:50:25.600956 18202 ts_tablet_manager.cc:1397] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.011s
I20260929 17:50:25.602113 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.584925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:36555 (local address 127.3.183.254:46883)
0929 17:50:25.585264 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.585282 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.586398 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 17:50:25.587500 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.587512 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.587859 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.588178 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:25.589299 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.590261 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.591396 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.591919 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.592082 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.596390 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:25.596397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:25.596413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:25.596446 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:25.598570 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.599195 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.599200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.599206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.599359 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.599770 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.599773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.599802 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.600248 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:25.600369 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.600998 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.601316 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
I20260929 17:50:25.602123 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.584925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:36555)
0929 17:50:25.586216 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.586260 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:25.587069 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:25.588254 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:25.588263 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:25.588675 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:25.589137 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.589151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.590411 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.590414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.591272 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.591282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.592053 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.592057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.592243 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.594167 (+ 1924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:25.594189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:25.596148 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:25.598703 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.598709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.598724 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.599070 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.599472 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.599475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.599477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.599660 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.600374 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:25.600381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:25.600674 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.600860 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.601211 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":136,"threads_started":1}
I20260929 17:50:25.603628 18202 raft_consensus.cc:348] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.604080 18202 raft_consensus.cc:729] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fcbfb2e9fc043dea44ceaf18cfe4f93, State: Initialized, Role: FOLLOWER
I20260929 17:50:25.604568 18202 consensus_queue.cc:260] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } }
I20260929 17:50:25.605428 18310 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46883
I20260929 17:50:25.605500 18202 ts_tablet_manager.cc:1428] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 17:50:25.605650 18310 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:25.606101 18310 heartbeater.cc:511] Master 127.3.183.254:46883 requested a full tablet report, sending...
I20260929 17:50:25.608376 18122 ts_manager.cc:194] Registered new tserver with Master: 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523)
I20260929 17:50:25.611732 18122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:36555
I20260929 17:50:25.614341 18310 heartbeater.cc:503] Master 127.3.183.254:46883 was elected leader, sending a full tablet report...
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:50:25.621721 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.611172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42523 (local address 127.0.0.1:40026)
0929 17:50:25.612076 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.612126 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:25.612253 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:25.612968 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:25.612974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:25.613393 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:25.613521 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.613529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.614773 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.614777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.615686 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.615696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.616391 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.616395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.616508 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.617022 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:25.617033 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:25.618380 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:25.620064 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.620073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.620078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.620307 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.620579 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.620582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.620584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.620664 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.621098 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:25.621104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:25.621215 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.621327 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.621542 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":118,"threads_started":1}
I20260929 17:50:25.621721 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.611264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.3.183.193:42523)
0929 17:50:25.612518 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.612529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.612550 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:50:25.612614 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.612619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.612811 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.612939 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:25.613660 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.614670 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.615866 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.616270 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.616367 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.618518 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:25.618528 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:25.618539 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:25.618560 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:25.619969 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.620411 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.620417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.620418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.620499 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.620741 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.620744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.620746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.620994 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:25.621105 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.621422 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.621555 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":109,"threads_started":1}
I20260929 17:50:25.622867 18244 tablet_service.cc:1514] Processing DeleteTablet for tablet 74cbf018a64a4789816f56d3fb8ccfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:25 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:40026
I20260929 17:50:25.623852 18319 tablet_replica.cc:330] stopping tablet replica
I20260929 17:50:25.624262 18319 raft_consensus.cc:2229] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:50:25.624531 18319 raft_consensus.cc:2258] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:25.625592 18319 ts_tablet_manager.cc:1905] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:50:25.630038 18319 ts_tablet_manager.cc:1918] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:50:25.630142 18319 log.cc:1196] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/74cbf018a64a4789816f56d3fb8ccfb1
I20260929 17:50:25.630396 18319 ts_tablet_manager.cc:1939] T 74cbf018a64a4789816f56d3fb8ccfb1 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Deleting consensus metadata
I20260929 17:50:25.632300 18107 catalog_manager.cc:4841] TS 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523): tablet 74cbf018a64a4789816f56d3fb8ccfb1 (table test-table [id=82e932dc10e64744925665c2ea482b06]) successfully deleted
W20260929 17:50:25.705391 18314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:25.705641 18314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:25.705685 18314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:25.709758 18314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:25.709826 18314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:25.709895 18314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:50:25.715399 18314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:42193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=43977
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:25.716524 18314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:25.717548 18314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:25.727078 18324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:25.727078 18327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:25.727196 18314 server_base.cc:1034] running on GCE node
W20260929 17:50:25.727102 18325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:25.727743 18314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:25.728415 18314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:25.729574 18314 hybrid_clock.cc:648] HybridClock initialized: now 1790704225729556 us; error 29 us; skew 500 ppm
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:25.732133 18314 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:50:25.732956 18314 webserver.cc:460] Webserver started at http://127.3.183.194:43977/ using document root <none> and password file <none>
I20260929 17:50:25.733474 18314 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:25.733554 18314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:25.737403 18314 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:50:25.739638 18334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.740958 18314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:50:25.741077 18314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "a5583a1f83704ef7b2290a51c639c374"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "08833794a352528c877c7be7b96d92e3"
server_key_iv: "f922e1291201861b52b1aa789f62b3ed"
server_key_version: "encryptionkey@0"
I20260929 17:50:25.741602 18314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:25.765244 18314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:25.768383 18314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:25.768567 18314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:25.769146 18314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:25.769960 18314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:25.770015 18314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.770068 18314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:25.770092 18314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.778520 18314 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:42193
I20260929 17:50:25.778556 18447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:42193 every 8 connection(s)
I20260929 17:50:25.779515 18314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:50:25.786544 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18314
I20260929 17:50:25.786995 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:39517
--local_ip_for_outbound_sockets=127.3.183.195
--tserver_master_addrs=127.3.183.254:46883
--webserver_port=46813
--webserver_interface=127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:25.793110 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.781360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:39971 (local address 127.3.183.254:46883)
0929 17:50:25.781511 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.781516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.782454 (+ 938us) server_negotiation.cc:407] Connection header received
0929 17:50:25.783699 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.783701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.783753 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.783860 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:25.785058 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.785582 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.786876 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.787131 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.787231 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.789777 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:25.789780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:25.789782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:25.789807 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:25.791095 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.791559 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.791563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.791564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.791641 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.791992 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.791996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.791997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.792187 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:25.792271 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.792791 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.792930 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:50:25.793924 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.781800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:39971)
0929 17:50:25.782264 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.782307 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:25.783514 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:25.784016 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:25.784023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:25.784437 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:25.784930 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.784944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.785695 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.785698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.786764 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.786776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.787288 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.787292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.787454 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.788032 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:25.788048 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:25.789615 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:25.791199 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.791205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.791221 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.791466 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.791719 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.791722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.791724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.791903 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.792283 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:25.792290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:25.792565 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.792734 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.793069 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260929 17:50:25.795051 18448 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46883
I20260929 17:50:25.795326 18448 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:25.795964 18448 heartbeater.cc:511] Master 127.3.183.254:46883 requested a full tablet report, sending...
I20260929 17:50:25.797077 18122 ts_manager.cc:194] Registered new tserver with Master: a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193)
I20260929 17:50:25.797825 18122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:39971
W20260929 17:50:25.898746 18451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:25.899012 18451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:25.899055 18451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:25.903026 18451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:25.903085 18451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:25.903149 18451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:50:25.908320 18451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:39517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=46813
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:25.909467 18451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:25.910552 18451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:25.919847 18456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:25.919858 18457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:25.919858 18459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:25.920073 18451 server_base.cc:1034] running on GCE node
I20260929 17:50:25.920435 18451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:25.921016 18451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:25.922165 18451 hybrid_clock.cc:648] HybridClock initialized: now 1790704225922139 us; error 38 us; skew 500 ppm
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:25.925030 18451 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:50:25.926014 18451 webserver.cc:460] Webserver started at http://127.3.183.195:46813/ using document root <none> and password file <none>
I20260929 17:50:25.926576 18451 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:25.926671 18451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:25.930609 18451 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:50:25.932626 18466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.933859 18451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:50:25.933977 18451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "db73a96a78d94374b3b5aacff24aab9b"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "4fb92ccde4b74c3c3be89781cdf3c985"
server_key_iv: "2e31344f29ee75e6e766336b71238f67"
server_key_version: "encryptionkey@0"
I20260929 17:50:25.934520 18451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:25.959980 18451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:25.963075 18451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:25.963260 18451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:25.963883 18451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:25.964705 18451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:25.964753 18451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.964805 18451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:25.964833 18451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:25.973078 18451 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:39517
I20260929 17:50:25.973117 18579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:39517 every 8 connection(s)
I20260929 17:50:25.974104 18451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:50:25 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:25.983744 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18451
I20260929 17:50:25.988072 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.975896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:37825 (local address 127.3.183.254:46883)
0929 17:50:25.976043 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.976047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.976920 (+ 873us) server_negotiation.cc:407] Connection header received
0929 17:50:25.978196 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.978199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.978261 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.978364 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:25.979566 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.980164 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.981356 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.981591 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.981691 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.984299 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:25.984304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:25.984307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:25.984339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:25.985991 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.986544 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.986547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.986549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.986634 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.986998 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.987003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.987004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.987221 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:25.987330 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.987803 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.987930 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:50:25.989012 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.976291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:37825)
0929 17:50:25.976739 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:25.976783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:25.977980 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:25.978550 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:25.978556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:25.978930 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:25.979414 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.979427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.980294 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.980297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.981220 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:25.981230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.981743 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.981747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:25.981925 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.982450 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:25.982468 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:25.984128 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:25.986104 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.986108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.986121 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.986447 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.986714 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:25.986717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:25.986718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:25.986883 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:25.987321 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:25.987325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:25.987556 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 17:50:25.987712 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 17:50:25.988032 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":93,"threads_started":1}
I20260929 17:50:25.990154 18580 heartbeater.cc:347] Connected to a master server at 127.3.183.254:46883
I20260929 17:50:25.990399 18580 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:25.990887 18580 heartbeater.cc:511] Master 127.3.183.254:46883 requested a full tablet report, sending...
I20260929 17:50:25.992039 18122 ts_manager.cc:194] Registered new tserver with Master: db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517)
I20260929 17:50:25.992669 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.984447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.3.183.254:46883)
0929 17:50:25.984670 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.984674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.984965 (+ 291us) server_negotiation.cc:407] Connection header received
0929 17:50:25.985082 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.985086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.985135 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.985216 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:25.986222 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.986933 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.987980 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.988287 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.988388 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.989602 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:25.989606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:25.989608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:25.989632 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:25.990958 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.991418 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.991421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.991423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.991510 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:25.991817 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:25.991821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:25.991823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:25.992022 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:25.992230 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.992355 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.992514 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260929 17:50:25.992739 18122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:37825
I20260929 17:50:25.994361 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:50:25.998413 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:25.994912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.3.183.254:46883)
0929 17:50:25.995025 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:25.995027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:25.995198 (+ 171us) server_negotiation.cc:407] Connection header received
0929 17:50:25.995312 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:25.995314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:25.995395 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:25.995468 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:50:25.995686 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.996251 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.997141 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:25.997353 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:25.997424 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:25.997729 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:50:25.998072 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:50:25.998158 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:25.998270 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 17:50:25.998319 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:50:25.999048 18121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:50:26.002416 18121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:50:26.010622 18244 tablet_service.cc:1467] Processing CreateTablet for tablet 82c29d03cc834a12a1cc403913cd64a0 (DEFAULT_TABLE table=test-table [id=66f6410140204ce0a7fb77a861267bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:26.011287 18244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c29d03cc834a12a1cc403913cd64a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:26.014140 18202 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap starting.
I20260929 17:50:26.015141 18202 tablet_bootstrap.cc:654] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:26.015208 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.009199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39517 (local address 127.0.0.1:42240)
0929 17:50:26.009315 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.009332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:26.009409 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:26.011195 (+ 1786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:26.011198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:26.011215 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:26.011293 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.011299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.012683 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.012687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.013722 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.013729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.014736 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.014739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.014852 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.014897 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 17:50:26.014984 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:50:26.015071 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:50:26.015283 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.009395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.3.183.195:39517)
0929 17:50:26.009781 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:26.009794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:26.009961 (+ 167us) server_negotiation.cc:407] Connection header received
0929 17:50:26.010047 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:26.010053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:26.011052 (+ 999us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:26.011215 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:26.011450 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.012541 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.013841 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.014606 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.014710 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.014859 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:26.015035 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 17:50:26.015141 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":185,"threads_started":1}
I20260929 17:50:26.015338 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.009435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42193 (local address 127.0.0.1:45410)
0929 17:50:26.009701 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.009715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:26.009801 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:26.010783 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:26.010786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:26.010807 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:26.010913 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.010919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.012242 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.012245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.013441 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.013457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.015013 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.015017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.015095 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.015115 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:50:26.015173 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 17:50:26.015236 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":126,"threads_started":1}
I20260929 17:50:26.015733 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.009961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.3.183.194:42193)
0929 17:50:26.010307 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:26.010319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:26.010356 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:50:26.010433 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:26.010439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:26.010639 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:26.010765 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:26.011029 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.012099 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.013596 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.014728 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.014819 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.014955 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:26.015485 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 17:50:26.015600 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":149,"threads_started":1}
I20260929 17:50:26.016709 18202 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: No bootstrap required, opened a new log
I20260929 17:50:26.016801 18202 ts_tablet_manager.cc:1397] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 17:50:26.017287 18202 raft_consensus.cc:348] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.017401 18202 raft_consensus.cc:374] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:26.017436 18202 raft_consensus.cc:729] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fcbfb2e9fc043dea44ceaf18cfe4f93, State: Initialized, Role: FOLLOWER
I20260929 17:50:26.017570 18202 consensus_queue.cc:260] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.017838 18202 ts_tablet_manager.cc:1428] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:50:26.018344 18514 tablet_service.cc:1467] Processing CreateTablet for tablet 82c29d03cc834a12a1cc403913cd64a0 (DEFAULT_TABLE table=test-table [id=66f6410140204ce0a7fb77a861267bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:26.018544 18382 tablet_service.cc:1467] Processing CreateTablet for tablet 82c29d03cc834a12a1cc403913cd64a0 (DEFAULT_TABLE table=test-table [id=66f6410140204ce0a7fb77a861267bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:26.019384 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c29d03cc834a12a1cc403913cd64a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:26.019409 18514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c29d03cc834a12a1cc403913cd64a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:26.026292 18594 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Bootstrap starting.
I20260929 17:50:26.028136 18593 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Bootstrap starting.
I20260929 17:50:26.028909 18594 tablet_bootstrap.cc:654] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:26.029804 18594 log.cc:824] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:26.030692 18593 tablet_bootstrap.cc:654] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:26.031483 18593 log.cc:824] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:26.032091 18594 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: No bootstrap required, opened a new log
I20260929 17:50:26.032269 18594 ts_tablet_manager.cc:1397] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:50:26.033545 18593 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: No bootstrap required, opened a new log
I20260929 17:50:26.033723 18593 ts_tablet_manager.cc:1397] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 17:50:26.036218 18594 raft_consensus.cc:348] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.036530 18594 raft_consensus.cc:374] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:26.036589 18594 raft_consensus.cc:729] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db73a96a78d94374b3b5aacff24aab9b, State: Initialized, Role: FOLLOWER
I20260929 17:50:26.037076 18594 consensus_queue.cc:260] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.037516 18593 raft_consensus.cc:348] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.037777 18580 heartbeater.cc:503] Master 127.3.183.254:46883 was elected leader, sending a full tablet report...
I20260929 17:50:26.037827 18593 raft_consensus.cc:374] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:26.037889 18593 raft_consensus.cc:729] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5583a1f83704ef7b2290a51c639c374, State: Initialized, Role: FOLLOWER
I20260929 17:50:26.038218 18594 ts_tablet_manager.cc:1428] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:50:26.038339 18593 consensus_queue.cc:260] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.039059 18448 heartbeater.cc:503] Master 127.3.183.254:46883 was elected leader, sending a full tablet report...
I20260929 17:50:26.039270 18593 ts_tablet_manager.cc:1428] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260929 17:50:26.083814 18311 tablet.cc:2369] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:50:26.225180 18581 tablet.cc:2369] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:50:26.280632 18449 tablet.cc:2369] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:50:26.331772 18597 raft_consensus.cc:482] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:50:26.332012 18597 raft_consensus.cc:504] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.333094 18597 leader_election.cc:288] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193), 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523)
I20260929 17:50:26.337352 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.333468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:36429 (local address 127.3.183.194:42193)
0929 17:50:26.333596 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:26.333602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:26.333617 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:50:26.333714 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:26.333717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:26.333765 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:26.333831 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:26.334204 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.334872 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.336276 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.336791 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.336908 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.336959 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:26.337157 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 17:50:26.337233 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:50:26.337337 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.333371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42193 (local address 127.3.183.195:36429)
0929 17:50:26.333482 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.333495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:26.333605 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:26.333953 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:26.333956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:26.333979 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:26.334096 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.334102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.334972 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.334974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.336167 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.336178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.336920 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.336923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.337029 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.337050 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:50:26.337130 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:50:26.337192 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":25}
I20260929 17:50:26.337918 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.333815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:54971 (local address 127.3.183.193:42523)
0929 17:50:26.334048 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:26.334051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:26.334064 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:50:26.334106 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:26.334108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:26.334159 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:26.334261 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:26.334589 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.335212 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.336536 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.337136 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.337248 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.337405 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:26.337742 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 17:50:26.337819 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260929 17:50:26.337929 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.333669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42523 (local address 127.3.183.195:54971)
0929 17:50:26.333934 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.333946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:26.334038 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:26.334280 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:26.334282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:26.334303 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:26.334477 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.334485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.335311 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.335314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.336438 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:26.336446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.337515 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.337522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:26.337616 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.337639 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:50:26.337729 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 17:50:26.337797 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260929 17:50:26.338027 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c29d03cc834a12a1cc403913cd64a0" candidate_uuid: "db73a96a78d94374b3b5aacff24aab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5583a1f83704ef7b2290a51c639c374" is_pre_election: true
I20260929 17:50:26.338300 18402 raft_consensus.cc:2454] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db73a96a78d94374b3b5aacff24aab9b in term 0.
I20260929 17:50:26.338759 18468 leader_election.cc:302] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5583a1f83704ef7b2290a51c639c374, db73a96a78d94374b3b5aacff24aab9b; no voters:
I20260929 17:50:26.338706 18264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c29d03cc834a12a1cc403913cd64a0" candidate_uuid: "db73a96a78d94374b3b5aacff24aab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" is_pre_election: true
I20260929 17:50:26.339022 18264 raft_consensus.cc:2454] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db73a96a78d94374b3b5aacff24aab9b in term 0.
I20260929 17:50:26.339032 18597 raft_consensus.cc:2781] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:50:26.339107 18597 raft_consensus.cc:482] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:50:26.339149 18597 raft_consensus.cc:3037] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:26.340273 18597 raft_consensus.cc:504] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.340575 18597 leader_election.cc:288] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [CANDIDATE]: Term 1 election: Requested vote from peers a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193), 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523)
I20260929 17:50:26.340911 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c29d03cc834a12a1cc403913cd64a0" candidate_uuid: "db73a96a78d94374b3b5aacff24aab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5583a1f83704ef7b2290a51c639c374"
I20260929 17:50:26.340947 18264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c29d03cc834a12a1cc403913cd64a0" candidate_uuid: "db73a96a78d94374b3b5aacff24aab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93"
I20260929 17:50:26.341061 18264 raft_consensus.cc:3037] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:26.341041 18402 raft_consensus.cc:3037] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:26.341964 18264 raft_consensus.cc:2454] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db73a96a78d94374b3b5aacff24aab9b in term 1.
I20260929 17:50:26.342042 18402 raft_consensus.cc:2454] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db73a96a78d94374b3b5aacff24aab9b in term 1.
I20260929 17:50:26.342209 18468 leader_election.cc:302] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fcbfb2e9fc043dea44ceaf18cfe4f93, db73a96a78d94374b3b5aacff24aab9b; no voters:
I20260929 17:50:26.342373 18597 raft_consensus.cc:2781] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:26.342572 18597 raft_consensus.cc:686] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 1 LEADER]: Becoming Leader. State: Replica: db73a96a78d94374b3b5aacff24aab9b, State: Running, Role: LEADER
I20260929 17:50:26.342897 18597 consensus_queue.cc:237] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.345944 18122 catalog_manager.cc:5495] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b reported cstate change: term changed from 0 to 1, leader changed from <none> to db73a96a78d94374b3b5aacff24aab9b (127.3.183.195). New cstate: current_term: 1 leader_uuid: "db73a96a78d94374b3b5aacff24aab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } health_report { overall_health: UNKNOWN } } }
I20260929 17:50:26.380789 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.377696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.3.183.195:39517)
0929 17:50:26.377816 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:26.377820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:26.377834 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:50:26.377907 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:26.377909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:26.377963 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:26.378061 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:50:26.378254 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.378778 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.379546 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:26.379753 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:26.379845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:26.380095 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:50:26.380491 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:50:26.380604 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:26.380630 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 17:50:26.380680 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:50:26.386381 18402 raft_consensus.cc:1261] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Refusing update from remote peer db73a96a78d94374b3b5aacff24aab9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:26.386389 18264 raft_consensus.cc:1261] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Refusing update from remote peer db73a96a78d94374b3b5aacff24aab9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:26.387048 18597 consensus_queue.cc:1035] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:26.387238 18601 consensus_queue.cc:1035] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:26.393302 18604 mvcc.cc:205] Tried to move back new op lower bound from 7334724511271505920 to 7334724511102767104. Current Snapshot: MvccSnapshot[applied={T|T < 7334724511271505920}]
I20260929 17:50:26.396996 18606 mvcc.cc:205] Tried to move back new op lower bound from 7334724511271505920 to 7334724511102767104. Current Snapshot: MvccSnapshot[applied={T|T < 7334724511271505920}]
I20260929 17:50:26.402751 18121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:26.402900 18121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:26.405120 18121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Sending DeleteTablet for 3 replicas of tablet 82c29d03cc834a12a1cc403913cd64a0
I20260929 17:50:26.405705 18514 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c29d03cc834a12a1cc403913cd64a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:26 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42240
I20260929 17:50:26.405855 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 18160
I20260929 17:50:26.405848 18382 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c29d03cc834a12a1cc403913cd64a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:26 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45410
I20260929 17:50:26.406538 18613 tablet_replica.cc:330] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: stopping tablet replica
I20260929 17:50:26.406868 18613 raft_consensus.cc:2229] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:50:26.407204 18613 raft_consensus.cc:2258] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:26.408694 18613 ts_tablet_manager.cc:1905] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:50:26.411356 18613 ts_tablet_manager.cc:1918] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:50:26.411449 18613 log.cc:1196] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/82c29d03cc834a12a1cc403913cd64a0
W20260929 17:50:26.411798 18107 connection.cc:535] client connection to 127.3.183.193:42523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:50:26.411826 18613 ts_tablet_manager.cc:1939] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Deleting consensus metadata
W20260929 17:50:26.411937 18107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 17:50:26.412827 18589 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:50:26.412583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42523 (local address 127.0.0.1:40042)
0929 17:50:26.412691 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.412751 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:42523: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 17:50:26.413071 18107 catalog_manager.cc:4584] TS 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82c29d03cc834a12a1cc403913cd64a0: Network error: Client connection negotiation failed: client connection to 127.3.183.193:42523: connect: Connection reset by peer (error 104)
I20260929 17:50:26.413270 18107 catalog_manager.cc:4841] TS db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517): tablet 82c29d03cc834a12a1cc403913cd64a0 (table test-table [id=66f6410140204ce0a7fb77a861267bee]) successfully deleted
I20260929 17:50:26.413350 18614 tablet_replica.cc:330] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: stopping tablet replica
I20260929 17:50:26.413640 18614 raft_consensus.cc:2229] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:50:26.413918 18614 raft_consensus.cc:2258] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:26.414294 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 18314
I20260929 17:50:26.419991 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 18451
I20260929 17:50:26.420966 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.420679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42193 (local address 127.0.0.1:45426)
0929 17:50:26.420837 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.420885 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 17:50:26.425019 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 18089
I20260929 17:50:26.430552 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46883
--webserver_interface=127.3.183.254
--webserver_port=42413
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:26.538358 18616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:26.538650 18616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:26.538689 18616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:26.542721 18616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:26.542790 18616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:26.542809 18616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:26.542824 18616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:26.542837 18616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:26.548137 18616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=42413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:26.549208 18616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:26.550244 18616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:50:26.558559 18616 server_base.cc:1034] running on GCE node
W20260929 17:50:26.558566 18622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:26.558538 18624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:26.558523 18621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:26.559424 18616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:26.560597 18616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:26.561782 18616 hybrid_clock.cc:648] HybridClock initialized: now 1790704226561746 us; error 50 us; skew 500 ppm
Sep 29 17:50:26 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:26.564554 18616 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:26.565419 18616 webserver.cc:460] Webserver started at http://127.3.183.254:42413/ using document root <none> and password file <none>
I20260929 17:50:26.565980 18616 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:26.566045 18616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:26.569975 18616 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:50:26.572232 18631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:26.573573 18616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:50:26.573690 18616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d35b618f759d2bcc80905aab8bf63092"
server_key_iv: "c62340d74ba2d53fb86d65c5f52bf2dc"
server_key_version: "encryptionkey@0"
I20260929 17:50:26.574265 18616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:26.595023 18616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:26.598007 18616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:26.598162 18616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:26.604709 18616 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:46883
I20260929 17:50:26.604960 18683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:46883 every 8 connection(s)
I20260929 17:50:26.605839 18616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:26.606499 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18616
I20260929 17:50:26.606928 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:42523
--local_ip_for_outbound_sockets=127.3.183.193
--tserver_master_addrs=127.3.183.254:46883
--webserver_port=45821
--webserver_interface=127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:26.610605 18684 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:50:26.612318 18684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Bootstrap starting.
I20260929 17:50:26.628980 18684 log.cc:824] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:26.639672 18684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:50:26.640045 18684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4: Bootstrap complete.
I20260929 17:50:26.643889 18684 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:26.644281 18684 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc89abc8edf94fa4ad37c3f4e327a8c4, State: Initialized, Role: FOLLOWER
I20260929 17:50:26.644765 18684 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:26.644886 18684 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:50:26.644951 18684 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:50:26.645037 18684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:50:26.646883 18684 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:26.647181 18684 leader_election.cc:302] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc89abc8edf94fa4ad37c3f4e327a8c4; no voters:
I20260929 17:50:26.647470 18684 leader_election.cc:288] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 17:50:26.647567 18688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 17:50:26.647749 18688 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [term 3 LEADER]: Becoming Leader. State: Replica: fc89abc8edf94fa4ad37c3f4e327a8c4, State: Running, Role: LEADER
I20260929 17:50:26.648069 18688 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } }
I20260929 17:50:26.648689 18684 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:26.648991 18690 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc89abc8edf94fa4ad37c3f4e327a8c4. Latest consensus state: current_term: 3 leader_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } } }
I20260929 17:50:26.649113 18690 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:26.649413 18689 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc89abc8edf94fa4ad37c3f4e327a8c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 46883 } } }
I20260929 17:50:26.649514 18689 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc89abc8edf94fa4ad37c3f4e327a8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:26.649991 18692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:26.654919 18692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:26.655730 18692 catalog_manager.cc:1259] Loaded cluster ID: 9a29fbd0c1cd4659a5c41a8c43aa5665
I20260929 17:50:26.655817 18692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 17:50:26.656819 18692 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9a91bc9ad google::LogMessage::Fail() at ??:0
@ 0x7fb9a91bf88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9a91bc4de google::LogMessage::Flush() at ??:0
@ 0x7fb9a91bda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9b3afbfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb9b3afcd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9b3af9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb9b3b3d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb9ab40e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9ab40ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb9ab4103f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb9ab3ff512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9aca436db start_thread at ??:0
@ 0x7fb9a7cba61f clone at ??:0
W20260929 17:50:26.727829 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:26.728087 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:26.728147 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:26.732184 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:26.732234 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:26.732311 18686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:50:26.737679 18686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:42523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=45821
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:26.738716 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:26.739722 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:26.749434 18725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:26.749498 18686 server_base.cc:1034] running on GCE node
W20260929 17:50:26.749677 18726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:26.749426 18728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:26.750207 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:26.750844 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:26.752010 18686 hybrid_clock.cc:648] HybridClock initialized: now 1790704226751983 us; error 39 us; skew 500 ppm
Sep 29 17:50:26 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:26.754647 18686 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:50:26.755565 18686 webserver.cc:460] Webserver started at http://127.3.183.193:45821/ using document root <none> and password file <none>
I20260929 17:50:26.756141 18686 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:26.756201 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:26.759958 18686 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:50:26.761989 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:26.763199 18686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:50:26.763293 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "d2497be67de89701e86d0cf70b588fd3"
server_key_iv: "90479cb556a18a52a539b647fab4e903"
server_key_version: "encryptionkey@0"
I20260929 17:50:26.763850 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:26.783490 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:26.786556 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:26.786731 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:26.787318 18686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:26.788434 18742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:50:26.792441 18686 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:50:26.792506 18686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 17:50:26.792558 18686 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:50:26.794256 18686 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:50:26.794312 18686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:50:26.794600 18742 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap starting.
I20260929 17:50:26.804421 18686 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:42523
I20260929 17:50:26.804715 18849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:42523 every 8 connection(s)
I20260929 17:50:26.805650 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:50:26.813608 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18686
I20260929 17:50:26.814114 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:42193
--local_ip_for_outbound_sockets=127.3.183.194
--tserver_master_addrs=127.3.183.254:46883
--webserver_port=43977
--webserver_interface=127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:26.814322 18742 log.cc:824] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:26.824177 18742 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:50:26.824620 18742 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Bootstrap complete.
I20260929 17:50:26.825973 18742 ts_tablet_manager.cc:1397] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.004s
I20260929 17:50:26.829387 18742 raft_consensus.cc:348] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.829990 18742 raft_consensus.cc:729] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fcbfb2e9fc043dea44ceaf18cfe4f93, State: Initialized, Role: FOLLOWER
I20260929 17:50:26.830540 18742 consensus_queue.cc:260] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:26.831645 18742 ts_tablet_manager.cc:1428] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
W20260929 17:50:26.867357 18852 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:50:26.808035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:60507)
0929 17:50:26.808613 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.808672 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:26.809697 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:26.866537 (+ 56840us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":144,"threads_started":1}
W20260929 17:50:26.867827 18850 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:46883 (0 consecutive failures): Network error: Failed to ping master at 127.3.183.254:46883: Client connection negotiation failed: client connection to 127.3.183.254:46883: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:50:26.868588 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.868298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:49813)
0929 17:50:26.868450 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.868533 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 17:50:26.869161 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:26.868991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:35493)
0929 17:50:26.869078 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:26.869117 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:50:26.869295 18850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:50:26.929141 18854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:26.929394 18854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:26.929433 18854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:26.933439 18854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:26.933504 18854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:26.933561 18854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:50:26.938827 18854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:42193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=43977
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:26.939899 18854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:26.940915 18854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:50:26.950290 18854 server_base.cc:1034] running on GCE node
W20260929 17:50:26.950248 18863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:26.950245 18861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:26.950245 18860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:26.950904 18854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:26.951534 18854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:26.952716 18854 hybrid_clock.cc:648] HybridClock initialized: now 1790704226952695 us; error 37 us; skew 500 ppm
Sep 29 17:50:26 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:26.955417 18854 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:50:26.956274 18854 webserver.cc:460] Webserver started at http://127.3.183.194:43977/ using document root <none> and password file <none>
I20260929 17:50:26.956786 18854 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:26.956846 18854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:26.960579 18854 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:26.962841 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:26.964115 18854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:26.964221 18854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "a5583a1f83704ef7b2290a51c639c374"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "08833794a352528c877c7be7b96d92e3"
server_key_iv: "f922e1291201861b52b1aa789f62b3ed"
server_key_version: "encryptionkey@0"
I20260929 17:50:26.964756 18854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:26.984297 18854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:26.987490 18854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:26.987681 18854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:26.988317 18854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:26.989439 18877 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:50:26.993570 18854 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:50:26.993635 18854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 17:50:26.993700 18854 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:50:26.995350 18854 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:50:26.995395 18854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:50:26.995671 18877 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Bootstrap starting.
I20260929 17:50:27.004617 18854 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:42193
I20260929 17:50:27.004685 18984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:42193 every 8 connection(s)
I20260929 17:50:27.005576 18854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:50:27.009004 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.007625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:41205)
0929 17:50:27.008111 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.008237 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":145,"threads_started":1}
W20260929 17:50:27.009563 18985 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:46883 (0 consecutive failures): Network error: Failed to ping master at 127.3.183.254:46883: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
I20260929 17:50:27.010174 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.009955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:44863)
0929 17:50:27.010069 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.010120 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 17:50:27.010787 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.010599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:41045)
0929 17:50:27.010697 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.010737 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 17:50:27.010996 18985 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 17:50:27.014045 18877 log.cc:824] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:27.014247 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18854
I20260929 17:50:27.014629 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:39517
--local_ip_for_outbound_sockets=127.3.183.195
--tserver_master_addrs=127.3.183.254:46883
--webserver_port=46813
--webserver_interface=127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:50:27.021486 18877 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:50:27.021821 18877 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Bootstrap complete.
I20260929 17:50:27.022778 18877 ts_tablet_manager.cc:1397] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.004s
I20260929 17:50:27.025532 18877 raft_consensus.cc:348] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:27.025967 18877 raft_consensus.cc:729] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5583a1f83704ef7b2290a51c639c374, State: Initialized, Role: FOLLOWER
I20260929 17:50:27.026444 18877 consensus_queue.cc:260] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:27.027446 18877 ts_tablet_manager.cc:1428] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 17:50:27.027771 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.027434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:46301)
0929 17:50:27.027617 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.027684 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:50:27.028590 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.028356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:49451)
0929 17:50:27.028479 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.028530 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 17:50:27.125839 18989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:27.126106 18989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:27.126148 18989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:27.130091 18989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:27.130151 18989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:27.130208 18989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:50:27.135339 18989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:39517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=46813
--tserver_master_addrs=127.3.183.254:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:27.136409 18989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:27.137414 18989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:27.147017 18998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:27.146988 18995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:27.147190 18996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:27.147475 18989 server_base.cc:1034] running on GCE node
I20260929 17:50:27.147933 18989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:27.148602 18989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:27.149786 18989 hybrid_clock.cc:648] HybridClock initialized: now 1790704227149744 us; error 55 us; skew 500 ppm
Sep 29 17:50:27 dist-test-slave-wbpt krb5kdc[17540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:27.152472 18989 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:50:27.153326 18989 webserver.cc:460] Webserver started at http://127.3.183.195:46813/ using document root <none> and password file <none>
I20260929 17:50:27.153853 18989 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:27.153920 18989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:27.157874 18989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:50:27.159947 19005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:27.161257 18989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:50:27.161375 18989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "db73a96a78d94374b3b5aacff24aab9b"
format_stamp: "Formatted at 2026-09-29 17:50:24 on dist-test-slave-wbpt"
server_key: "4fb92ccde4b74c3c3be89781cdf3c985"
server_key_iv: "2e31344f29ee75e6e766336b71238f67"
server_key_version: "encryptionkey@0"
I20260929 17:50:27.161893 18989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:27.194479 18989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:27.197712 18989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:27.197892 18989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:27.198486 18989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:27.199311 18989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:27.199357 18989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:27.199409 18989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:27.199435 18989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:27.207845 18989 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:39517
I20260929 17:50:27.207862 19118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:39517 every 8 connection(s)
I20260929 17:50:27.208827 18989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:50:27.211870 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 18989
I20260929 17:50:27.213025 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.210902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:57501)
0929 17:50:27.211370 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.211519 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
W20260929 17:50:27.213445 19119 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:46883 (0 consecutive failures): Network error: Failed to ping master at 127.3.183.254:46883: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
I20260929 17:50:27.213986 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.213783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:51511)
0929 17:50:27.213884 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.213932 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 17:50:27.214555 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.214334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:56019)
0929 17:50:27.214457 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.214502 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 17:50:27.214699 19119 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 17:50:27.870304 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:27.869842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:58569)
0929 17:50:27.870105 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:27.870170 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260929 17:50:28.029897 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:28.029385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:42147)
0929 17:50:28.029705 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:28.029799 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260929 17:50:28.215725 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:28.215279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:58827)
0929 17:50:28.215571 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:28.215628 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":119,"threads_started":1}
I20260929 17:50:28.574060 19126 raft_consensus.cc:482] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:50:28.574255 19126 raft_consensus.cc:504] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:28.575392 19126 leader_election.cc:288] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517), a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193)
Sep 29 17:50:28 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.194@KRBTEST.COM
Sep 29 17:50:28 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.195@KRBTEST.COM
I20260929 17:50:28.589879 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:28.576049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42193 (local address 127.3.183.193:42607)
0929 17:50:28.576360 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:28.576374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:28.576465 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:28.577517 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:28.577523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:28.578046 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:28.578503 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:28.578518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.579805 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.579808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:28.580806 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:28.580816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.581518 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.581522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:28.581699 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:28.582317 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:28.582330 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:28.584082 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:28.586792 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:28.586795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:28.586796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:28.587050 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:28.587954 (+ 904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:28.587958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:28.587960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:28.588180 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:28.588917 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:28.588920 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:28.589126 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 17:50:28.589287 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 17:50:28.589628 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":86,"spinlock_wait_cycles":5504,"thread_start_us":65,"threads_started":1}
I20260929 17:50:28.589965 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:28.576256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:42607 (local address 127.3.183.194:42193)
0929 17:50:28.576564 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:28.576573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:28.576603 (+ 30us) server_negotiation.cc:407] Connection header received
0929 17:50:28.576778 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:28.576782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:28.577142 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:28.577395 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:28.578673 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.579622 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.580931 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.581396 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.581542 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:28.584307 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:28.584315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:28.584335 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:28.584370 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:28.586674 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:28.587291 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:28.587383 (+ 92us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:28.587399 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:28.587718 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:28.588325 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:28.588329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:28.588330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:28.588790 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:28.588931 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:28.589467 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 17:50:28.589767 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260929 17:50:28.590008 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:28.575808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:46831 (local address 127.3.183.195:39517)
0929 17:50:28.576239 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:28.576266 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:28.576313 (+ 47us) server_negotiation.cc:407] Connection header received
0929 17:50:28.577076 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:28.577099 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:28.577436 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:28.577741 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:28.578669 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.579744 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.580944 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.581401 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.581548 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:28.584569 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:28.584574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:28.584648 (+ 74us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:28.584686 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:28.586574 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:28.587610 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:28.587617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:28.587623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:28.587810 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:28.588293 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:28.588295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:28.588297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:28.588757 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:28.588868 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:28.589513 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 17:50:28.589842 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":73,"threads_started":1}
I20260929 17:50:28.589923 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:28.575659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:39517 (local address 127.3.183.193:46831)
0929 17:50:28.576056 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:28.576075 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:28.576195 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:28.577708 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:28.577711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:28.578044 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:28.578502 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:28.578518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.579893 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.579895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:28.580806 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:28.580816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:28.581566 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:28.581569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:28.581700 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:28.582288 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:28.582317 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:28.584353 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:28.586720 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:28.586726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:28.586742 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:28.587294 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:28.588091 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:28.588093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:28.588095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:28.588183 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:28.588883 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:28.588889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:28.589126 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 17:50:28.589376 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 17:50:28.589721 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":190,"spinlock_wait_cycles":28288,"thread_start_us":79,"threads_started":1}
I20260929 17:50:28.591557 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c29d03cc834a12a1cc403913cd64a0" candidate_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a5583a1f83704ef7b2290a51c639c374" is_pre_election: true
I20260929 17:50:28.591908 18939 raft_consensus.cc:2454] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fcbfb2e9fc043dea44ceaf18cfe4f93 in term 1.
W20260929 17:50:28.592911 18737 leader_election.cc:341] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517): Not found: Tablet not found: 82c29d03cc834a12a1cc403913cd64a0
I20260929 17:50:28.593153 18737 leader_election.cc:302] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6fcbfb2e9fc043dea44ceaf18cfe4f93, a5583a1f83704ef7b2290a51c639c374; no voters: db73a96a78d94374b3b5aacff24aab9b
I20260929 17:50:28.593485 19126 raft_consensus.cc:2781] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 17:50:28.593658 19126 raft_consensus.cc:482] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:50:28.593750 19126 raft_consensus.cc:3037] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:50:28.596160 19126 raft_consensus.cc:504] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:28.596611 19126 leader_election.cc:288] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 2 election: Requested vote from peers db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517), a5583a1f83704ef7b2290a51c639c374 (127.3.183.194:42193)
I20260929 17:50:28.597250 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c29d03cc834a12a1cc403913cd64a0" candidate_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a5583a1f83704ef7b2290a51c639c374"
W20260929 17:50:28.597416 18737 leader_election.cc:341] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517): Not found: Tablet not found: 82c29d03cc834a12a1cc403913cd64a0
I20260929 17:50:28.597442 18939 raft_consensus.cc:3037] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:50:28.599653 18939 raft_consensus.cc:2454] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fcbfb2e9fc043dea44ceaf18cfe4f93 in term 2.
I20260929 17:50:28.600134 18737 leader_election.cc:302] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6fcbfb2e9fc043dea44ceaf18cfe4f93, a5583a1f83704ef7b2290a51c639c374; no voters: db73a96a78d94374b3b5aacff24aab9b
I20260929 17:50:28.600302 19126 raft_consensus.cc:2781] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:50:28.600485 19126 raft_consensus.cc:686] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [term 2 LEADER]: Becoming Leader. State: Replica: 6fcbfb2e9fc043dea44ceaf18cfe4f93, State: Running, Role: LEADER
I20260929 17:50:28.600867 19126 consensus_queue.cc:237] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:28.601413 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:28.600982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:49137)
0929 17:50:28.601236 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:28.601297 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260929 17:50:29.031193 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.030752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:34759)
0929 17:50:29.031038 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.031101 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
W20260929 17:50:29.039443 18737 consensus_peers.cc:487] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 -> Peer db73a96a78d94374b3b5aacff24aab9b (127.3.183.195:39517): Couldn't send request to peer db73a96a78d94374b3b5aacff24aab9b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82c29d03cc834a12a1cc403913cd64a0. This is attempt 1: this message will repeat every 5th retry.
I20260929 17:50:29.173528 18939 raft_consensus.cc:1261] T 82c29d03cc834a12a1cc403913cd64a0 P a5583a1f83704ef7b2290a51c639c374 [term 2 FOLLOWER]: Refusing update from remote peer 6fcbfb2e9fc043dea44ceaf18cfe4f93: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 17:50:29.174378 19126 consensus_queue.cc:1035] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 17:50:29.176499 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.176203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:56591)
0929 17:50:29.176332 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.176407 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:50:29.179136 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.178800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:45351)
0929 17:50:29.178991 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.179042 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260929 17:50:29.179454 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.179258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:50321)
0929 17:50:29.179343 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.179393 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 17:50:29.216925 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.216514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:53909)
0929 17:50:29.216766 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.216824 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260929 17:50:29.460856 19141 ts_tablet_manager.cc:927] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Initiating tablet copy from peer 6fcbfb2e9fc043dea44ceaf18cfe4f93 (127.3.183.193:42523)
I20260929 17:50:29.461485 19141 tablet_copy_client.cc:316] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: tablet copy: Beginning tablet copy session from remote peer at address 127.3.183.193:42523
Sep 29 17:50:29 dist-test-slave-wbpt krb5kdc[17540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:50:29.471401 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.462022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:42523 (local address 127.3.183.195:60121)
0929 17:50:29.462210 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.462231 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:29.462359 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:29.462891 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:29.462896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:29.463255 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:29.463365 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:29.463371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:29.464408 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:29.464411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:29.465061 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:29.465070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:29.465680 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:29.465683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:29.465767 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:29.466358 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:29.466370 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:29.467872 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:29.469846 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:29.469854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:29.469859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:29.470091 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:29.470383 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:29.470385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:29.470387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:29.470462 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:29.470896 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:29.470902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:29.471026 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 17:50:29.471141 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 17:50:29.471274 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 17:50:29.471493 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.462188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:60121 (local address 127.3.183.193:42523)
0929 17:50:29.462484 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:29.462494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:29.462516 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:50:29.462566 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:29.462571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:29.462744 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:29.462856 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:29.463493 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:29.464287 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:29.465173 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:29.465568 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:29.465654 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:29.468060 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:29.468070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:29.468081 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:29.468108 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:29.469736 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:29.470207 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:29.470214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:29.470215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:29.470301 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:29.470549 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:29.470551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:29.470552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:29.470802 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:29.470905 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:29.471213 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 17:50:29.471338 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260929 17:50:29.472569 18824 tablet_copy_service.cc:140] P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Received BeginTabletCopySession request for tablet 82c29d03cc834a12a1cc403913cd64a0 from peer db73a96a78d94374b3b5aacff24aab9b ({username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:60121)
I20260929 17:50:29.472731 18824 tablet_copy_service.cc:161] P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Beginning new tablet copy session on tablet 82c29d03cc834a12a1cc403913cd64a0 from peer db73a96a78d94374b3b5aacff24aab9b at {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:60121: session id = db73a96a78d94374b3b5aacff24aab9b-82c29d03cc834a12a1cc403913cd64a0
I20260929 17:50:29.473948 18824 tablet_copy_source_session.cc:205] T 82c29d03cc834a12a1cc403913cd64a0 P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 17:50:29.477937 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c29d03cc834a12a1cc403913cd64a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:29.484292 19141 tablet_copy_client.cc:791] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: tablet copy: Starting download of 0 data blocks...
I20260929 17:50:29.484474 19141 tablet_copy_client.cc:655] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: tablet copy: Starting download of 1 WAL segments...
I20260929 17:50:29.486207 19141 tablet_copy_client.cc:531] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 17:50:29.487815 19141 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Bootstrap starting.
I20260929 17:50:29.493103 19141 log.cc:824] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:29.499508 19141 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:50:29.499755 19141 tablet_bootstrap.cc:492] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Bootstrap complete.
I20260929 17:50:29.500075 19141 ts_tablet_manager.cc:1397] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.004s
I20260929 17:50:29.502278 19141 raft_consensus.cc:348] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:29.502513 19141 raft_consensus.cc:729] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: db73a96a78d94374b3b5aacff24aab9b, State: Initialized, Role: FOLLOWER
I20260929 17:50:29.502877 19141 consensus_queue.cc:260] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db73a96a78d94374b3b5aacff24aab9b" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 39517 } } peers { permanent_uuid: "a5583a1f83704ef7b2290a51c639c374" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42193 } } peers { permanent_uuid: "6fcbfb2e9fc043dea44ceaf18cfe4f93" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 42523 } }
I20260929 17:50:29.503768 19141 ts_tablet_manager.cc:1428] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260929 17:50:29.504150 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.503878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:49541)
0929 17:50:29.504020 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.504084 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 17:50:29.504531 18824 tablet_copy_service.cc:342] P 6fcbfb2e9fc043dea44ceaf18cfe4f93: Request end of tablet copy session db73a96a78d94374b3b5aacff24aab9b-82c29d03cc834a12a1cc403913cd64a0 received from {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:60121
I20260929 17:50:29.504654 18824 tablet_copy_service.cc:433] P 6fcbfb2e9fc043dea44ceaf18cfe4f93: ending tablet copy session db73a96a78d94374b3b5aacff24aab9b-82c29d03cc834a12a1cc403913cd64a0 on tablet 82c29d03cc834a12a1cc403913cd64a0 with peer db73a96a78d94374b3b5aacff24aab9b
I20260929 17:50:29.504803 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.504592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:33933)
0929 17:50:29.504705 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.504753 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:50:29.990665 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.990259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:54607)
0929 17:50:29.990526 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.990572 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":52,"threads_started":1}
I20260929 17:50:29.990722 19073 raft_consensus.cc:1203] T 82c29d03cc834a12a1cc403913cd64a0 P db73a96a78d94374b3b5aacff24aab9b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 17:50:29.991618 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:29.991397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:52897)
0929 17:50:29.991504 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:29.991553 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 17:50:30.180605 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:30.180150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:39467)
0929 17:50:30.180446 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:30.180509 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260929 17:50:30.991846 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:30.991389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:53603)
0929 17:50:30.991648 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:30.991707 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260929 17:50:30.992616 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:30.992226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:51055)
0929 17:50:30.992485 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:30.992538 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":58,"threads_started":1}
I20260929 17:50:31.181803 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:31.181345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:35513)
0929 17:50:31.181632 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:31.181696 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
I20260929 17:50:31.993360 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:31.992884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:32809)
0929 17:50:31.993167 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:31.993230 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
I20260929 17:50:31.993695 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:31.993266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:48309)
0929 17:50:31.993551 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:31.993611 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260929 17:50:31.994145 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:31.993928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:58259)
0929 17:50:31.994036 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:31.994087 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 17:50:32.183133 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:32.182692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:46697)
0929 17:50:32.182970 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:32.183038 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260929 17:50:32.995373 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:32.994855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:56389)
0929 17:50:32.995225 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:32.995279 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":69,"threads_started":1}
I20260929 17:50:32.995528 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:32.994723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:38227)
0929 17:50:32.995311 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:32.995375 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":86,"threads_started":1}
I20260929 17:50:33.184501 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:33.183973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:40921)
0929 17:50:33.184340 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:33.184407 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":135,"threads_started":1}
I20260929 17:50:33.996769 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:33.996313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:55049)
0929 17:50:33.996616 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:33.996664 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":70,"threads_started":1}
I20260929 17:50:33.996769 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:33.996279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:32983)
0929 17:50:33.996597 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:33.996664 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260929 17:50:34.185763 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:34.185257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:58419)
0929 17:50:34.185607 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:34.185669 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260929 17:50:34.998072 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:34.997615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:56133)
0929 17:50:34.997892 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:34.997976 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260929 17:50:34.998087 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:34.997615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:53569)
0929 17:50:34.997950 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:34.998002 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":87,"threads_started":1}
I20260929 17:50:35.187112 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:35.186667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:44069)
0929 17:50:35.186948 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:35.187014 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260929 17:50:35.999409 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:35.998948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:54731)
0929 17:50:35.999248 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:35.999311 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260929 17:50:35.999466 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:35.998948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:60443)
0929 17:50:35.999312 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:35.999366 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":106,"threads_started":1}
I20260929 17:50:36.188375 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:36.187951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:51911)
0929 17:50:36.188228 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:36.188282 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260929 17:50:37.000712 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:37.000263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:37393)
0929 17:50:37.000578 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:37.000628 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":74,"threads_started":1}
I20260929 17:50:37.000712 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:37.000248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:54417)
0929 17:50:37.000533 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:37.000598 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260929 17:50:37.189622 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:37.189208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:57555)
0929 17:50:37.189467 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:37.189528 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260929 17:50:38.002033 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:38.001566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:37301)
0929 17:50:38.001847 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:38.001922 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260929 17:50:38.002060 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:38.001566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:44123)
0929 17:50:38.001918 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:38.001975 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":87,"threads_started":1}
I20260929 17:50:38.190840 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:38.190395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:37675)
0929 17:50:38.190665 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:38.190729 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260929 17:50:39.003333 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:39.002854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:43565)
0929 17:50:39.003201 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:39.003252 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":82,"threads_started":1}
I20260929 17:50:39.003307 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:39.002854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:35435)
0929 17:50:39.003115 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:39.003187 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260929 17:50:39.191995 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:39.191531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:48253)
0929 17:50:39.191827 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:39.191894 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260929 17:50:40.004514 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:40.004080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:40965)
0929 17:50:40.004357 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:40.004416 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260929 17:50:40.004513 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:40.004116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:37595)
0929 17:50:40.004369 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:40.004423 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260929 17:50:40.193228 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:40.192779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:35759)
0929 17:50:40.193073 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:40.193134 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
I20260929 17:50:41.005719 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:41.005307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:43487)
0929 17:50:41.005567 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:41.005625 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260929 17:50:41.005759 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:41.005307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:51737)
0929 17:50:41.005624 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:41.005673 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":78,"threads_started":1}
I20260929 17:50:41.194471 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:41.194050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:51293)
0929 17:50:41.194316 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:41.194376 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260929 17:50:42.007021 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:42.006635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.193:33799)
0929 17:50:42.006863 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:42.006934 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260929 17:50:42.007089 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:42.006597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.195:43085)
0929 17:50:42.006928 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:42.006988 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":85,"threads_started":1}
I20260929 17:50:42.195827 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:42.195394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:46883 (local address 127.3.183.194:49067)
0929 17:50:42.195647 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:42.195705 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260929 17:50:42.221743 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 18686
I20260929 17:50:42.226945 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 18854
I20260929 17:50:42.231985 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 18989
2026-09-29T17:50:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19259 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:50:42 dist-test-slave-wbpt krb5kdc[19191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:50:42 dist-test-slave-wbpt krb5kdc[19191](info): set up 2 sockets
Sep 29 17:50:42 dist-test-slave-wbpt krb5kdc[19191](info): commencing operation
krb5kdc: starting...
W20260929 17:50:43.195739 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:50:43 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:50:43.314493 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44201
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:43.423872 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:43.424152 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:43.424191 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:43.428200 19204 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 17:50:43.428259 19204 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 17:50:43.428273 19204 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 17:50:43.428287 19204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:50:43.428306 19204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:43.428323 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:43.428337 19204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:50:43.428351 19204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:50:43.433734 19204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:43.434808 19204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:43.435863 19204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:43.444234 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:43.444216 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:43.444263 19204 server_base.cc:1034] running on GCE node
W20260929 17:50:43.444306 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:43.445130 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:43.446203 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:43.447352 19204 hybrid_clock.cc:648] HybridClock initialized: now 1790704243447335 us; error 31 us; skew 500 ppm
Sep 29 17:50:43 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704243, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:43.450187 19204 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:50:43.451076 19204 webserver.cc:460] Webserver started at http://127.3.183.254:38655/ using document root <none> and password file <none>
I20260929 17:50:43.451604 19204 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:43.451661 19204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:43.451886 19204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:43.453886 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "4315ebb0b43a41309f23145e433fbaac"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "f02251f69bb3b250dfb51a742e2e392d"
server_key_iv: "02d078496d31289aa1583621a7610004"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.454413 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "4315ebb0b43a41309f23145e433fbaac"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "f02251f69bb3b250dfb51a742e2e392d"
server_key_iv: "02d078496d31289aa1583621a7610004"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.458029 19204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:50:43.460342 19219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:43.461529 19204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:50:43.461627 19204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "4315ebb0b43a41309f23145e433fbaac"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "f02251f69bb3b250dfb51a742e2e392d"
server_key_iv: "02d078496d31289aa1583621a7610004"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.461728 19204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:43.479820 19204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:43.482824 19204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:43.482987 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:43.489678 19204 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:44201
I20260929 17:50:43.490031 19271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:44201 every 8 connection(s)
I20260929 17:50:43.490651 19204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:50:43.493417 19272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:43.498742 19272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Bootstrap starting.
I20260929 17:50:43.500681 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 19204
I20260929 17:50:43.500783 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:50:43.501039 3807 external_mini_cluster.cc:1428] Setting key da087bdcb199987af59f305e04041307
I20260929 17:50:43.501286 19272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:43.502132 19272 log.cc:824] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:43.504218 19272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: No bootstrap required, opened a new log
I20260929 17:50:43.508402 19272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:50:43.508630 19272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:43.508673 19272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4315ebb0b43a41309f23145e433fbaac, State: Initialized, Role: FOLLOWER
I20260929 17:50:43.509099 19272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:50:43.509256 19272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:50:43.509318 19272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:50:43.509414 19272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 0 FOLLOWER]: Advancing to term 1
Sep 29 17:50:43 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:43.510366 19272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:50:43.510695 19272 leader_election.cc:302] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4315ebb0b43a41309f23145e433fbaac; no voters:
I20260929 17:50:43.511058 19272 leader_election.cc:288] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:50:43.511139 19277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:43.511595 19277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 LEADER]: Becoming Leader. State: Replica: 4315ebb0b43a41309f23145e433fbaac, State: Running, Role: LEADER
I20260929 17:50:43.511968 19277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:50:43.512413 19272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:50:43.513624 19278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4315ebb0b43a41309f23145e433fbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } } }
I20260929 17:50:43.513751 19278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:43.514060 19279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4315ebb0b43a41309f23145e433fbaac. Latest consensus state: current_term: 1 leader_uuid: "4315ebb0b43a41309f23145e433fbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } } }
I20260929 17:50:43.514163 19279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: This master's current role is: LEADER
I20260929 17:50:43.514276 19288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:50:43.516893 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:43.502069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.3.183.254:44201)
0929 17:50:43.502443 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:43.502458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:43.502506 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:50:43.503231 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:43.503251 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:43.503587 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:43.503910 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:43.504707 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.505677 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.506881 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.507401 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.507565 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:43.510345 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:43.510350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:43.510364 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:43.510391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:43.512279 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:43.512740 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:43.512746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:43.512752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:43.512865 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:43.514289 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:43.514293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:43.514294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:43.514797 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:43.514929 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:43.515823 (+ 894us) server_negotiation.cc:299] Negotiation successful
0929 17:50:43.516182 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260929 17:50:43.517611 19288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:50:43.522984 19288 catalog_manager.cc:1347] Generated new cluster ID: b7c29504a15f47329e7551ebf6c58197
I20260929 17:50:43.523067 19288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:50:43.536386 19288 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:50:43.537483 19288 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:50:43.548787 19288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Generated new TSK 0
I20260929 17:50:43.549556 19288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:50:43.579608 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:44201
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:43.689193 19298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:43.689476 19298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:43.689529 19298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:43.693579 19298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:43.693640 19298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:43.693707 19298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:50:43.699100 19298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:43.700201 19298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:43.701210 19298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:43.710765 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:43.710804 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:43.710944 19298 server_base.cc:1034] running on GCE node
W20260929 17:50:43.710777 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:43.711544 19298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:43.712189 19298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:43.713340 19298 hybrid_clock.cc:648] HybridClock initialized: now 1790704243713314 us; error 42 us; skew 500 ppm
Sep 29 17:50:43 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704243, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:43.716132 19298 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:50:43.717018 19298 webserver.cc:460] Webserver started at http://127.3.183.193:39753/ using document root <none> and password file <none>
I20260929 17:50:43.717568 19298 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:43.717631 19298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:43.717836 19298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:43.719897 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "a405a510e8ff49e7a5e7089746685b13"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "bb7b62f64061b7341d74f2053448e78a"
server_key_iv: "95c4d8eb9b49b33f7cd085b488364adb"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.720435 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "a405a510e8ff49e7a5e7089746685b13"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "bb7b62f64061b7341d74f2053448e78a"
server_key_iv: "95c4d8eb9b49b33f7cd085b488364adb"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.724104 19298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:50:43.726452 19313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:43.727643 19298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:50:43.727751 19298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "a405a510e8ff49e7a5e7089746685b13"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "bb7b62f64061b7341d74f2053448e78a"
server_key_iv: "95c4d8eb9b49b33f7cd085b488364adb"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.727886 19298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:43.742866 19298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:43.746068 19298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:43.746251 19298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:43.746855 19298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:43.747682 19298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:43.747730 19298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:43.747773 19298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:43.747835 19298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:43.756539 19298 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:36501
I20260929 17:50:43.756590 19426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:36501 every 8 connection(s)
I20260929 17:50:43.757557 19298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:50:43.765820 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 19298
I20260929 17:50:43.765928 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:50:43.766203 3807 external_mini_cluster.cc:1428] Setting key 915148dc6a4b9d1e375ed82f1e62cda0
Sep 29 17:50:43 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704243, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:43.770699 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:43.759327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:41911 (local address 127.3.183.254:44201)
0929 17:50:43.759494 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:43.759499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:43.760367 (+ 868us) server_negotiation.cc:407] Connection header received
0929 17:50:43.761605 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:43.761608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:43.761654 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:43.761724 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:43.762876 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.763407 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.764591 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.764819 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.764898 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:43.767361 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:43.767365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:43.767368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:43.767398 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:43.768762 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:43.769234 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:43.769237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:43.769238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:43.769314 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:43.769645 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:43.769649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:43.769650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:43.769829 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:43.769918 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:43.770413 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 17:50:43.770537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:50:43.771533 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:43.759728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:41911)
0929 17:50:43.760216 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:43.760260 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:43.761423 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:43.761872 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:43.761879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:43.762230 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:43.762759 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:43.762772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.763501 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.763503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:43.764476 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:43.764486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.764939 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.764941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:43.765094 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:43.765636 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:43.765656 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:43.767200 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:43.768873 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:43.768879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:43.768896 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:43.769139 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:43.769401 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:43.769404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:43.769406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:43.769551 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:43.769938 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:43.769944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:43.770187 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 17:50:43.770355 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 17:50:43.770680 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":116,"threads_started":1}
I20260929 17:50:43.772800 19427 heartbeater.cc:347] Connected to a master server at 127.3.183.254:44201
I20260929 17:50:43.773044 19427 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:43.773530 19427 heartbeater.cc:511] Master 127.3.183.254:44201 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:50:43.775108 19236 ts_manager.cc:194] Registered new tserver with Master: a405a510e8ff49e7a5e7089746685b13 (127.3.183.193:36501)
I20260929 17:50:43.776382 19236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:41911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:50:43.793329 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:44201
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:43.902484 19432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:43.902751 19432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:43.902794 19432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:43.906778 19432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:43.906841 19432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:43.906900 19432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:50:43.912326 19432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:43.913523 19432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:43.914624 19432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:43.924207 19437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:43.924214 19440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:43.924214 19438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:43.924364 19432 server_base.cc:1034] running on GCE node
I20260929 17:50:43.924914 19432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:43.925554 19432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:43.926716 19432 hybrid_clock.cc:648] HybridClock initialized: now 1790704243926698 us; error 29 us; skew 500 ppm
Sep 29 17:50:43 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704243, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:43.934782 19432 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:50:43.935762 19432 webserver.cc:460] Webserver started at http://127.3.183.194:34983/ using document root <none> and password file <none>
I20260929 17:50:43.936360 19432 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:43.936429 19432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:43.936631 19432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:43.938690 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "8499e8b033894eedbc118a4dd0072142"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "621621c4b56bf0a8841194c6d5ffc871"
server_key_iv: "85f8cda0d357a07810a47792fca24a40"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.939231 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "8499e8b033894eedbc118a4dd0072142"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "621621c4b56bf0a8841194c6d5ffc871"
server_key_iv: "85f8cda0d357a07810a47792fca24a40"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.942870 19432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 17:50:43.945258 19447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:43.946471 19432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:50:43.946587 19432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "8499e8b033894eedbc118a4dd0072142"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "621621c4b56bf0a8841194c6d5ffc871"
server_key_iv: "85f8cda0d357a07810a47792fca24a40"
server_key_version: "encryptionkey@0"
I20260929 17:50:43.946681 19432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:43.970813 19432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:43.974018 19432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:43.974205 19432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:43.974805 19432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:43.975631 19432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:43.975679 19432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:43.975731 19432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:43.975759 19432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:43.984292 19432 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:45043
I20260929 17:50:43.984336 19560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:45043 every 8 connection(s)
I20260929 17:50:43.985423 19432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:50:43.989825 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 19432
I20260929 17:50:43.989953 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:50:43.990278 3807 external_mini_cluster.cc:1428] Setting key 483c0bee9f41da82ae3bbeecffd5e25b
Sep 29 17:50:43 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704243, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:50:43.998740 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:43.987230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:57947 (local address 127.3.183.254:44201)
0929 17:50:43.987385 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:43.987389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:43.988452 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 17:50:43.989299 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:43.989302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:43.989350 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:43.989424 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:43.990938 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.991629 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.992811 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.993042 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.993129 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:43.995471 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:43.995475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:43.995478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:43.995505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:43.996808 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:43.997300 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:43.997303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:43.997304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:43.997377 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:43.997682 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:43.997685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:43.997686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:43.997881 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:43.997965 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:43.998446 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 17:50:43.998567 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:50:43.999589 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:43.987604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:57947)
0929 17:50:43.988271 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:43.988312 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:43.989111 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:43.989668 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:43.989678 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:43.990156 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:43.990792 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:43.990809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.991745 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.991748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:43.992695 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:43.992704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:43.993179 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:43.993184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:43.993340 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:43.993902 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:43.993918 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:43.995302 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:43.996928 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:43.996934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:43.996949 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:43.997209 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:43.997457 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:43.997460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:43.997462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:43.997599 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:43.997978 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:43.997985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:43.998228 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 17:50:43.998387 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 17:50:43.998730 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":132,"threads_started":1}
I20260929 17:50:44.000748 19561 heartbeater.cc:347] Connected to a master server at 127.3.183.254:44201
I20260929 17:50:44.000976 19561 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:50:44.001497 19561 heartbeater.cc:511] Master 127.3.183.254:44201 requested a full tablet report, sending...
I20260929 17:50:44.002561 19236 ts_manager.cc:194] Registered new tserver with Master: 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
I20260929 17:50:44.003273 19236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:57947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:50:44.018494 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:44201
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:50:44.128892 19566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:50:44.129171 19566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:50:44.129230 19566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:50:44.133395 19566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:50:44.133464 19566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:50:44.133554 19566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:50:44.138996 19566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:50:44.140153 19566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:50:44.141201 19566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:50:44.151357 19571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:44.151490 19566 server_base.cc:1034] running on GCE node
W20260929 17:50:44.151628 19574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:50:44.151635 19572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:50:44.152161 19566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:50:44.152837 19566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:50:44.154047 19566 hybrid_clock.cc:648] HybridClock initialized: now 1790704244154019 us; error 40 us; skew 500 ppm
Sep 29 17:50:44 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704244, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:50:44.157182 19566 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:50:44.158303 19566 webserver.cc:460] Webserver started at http://127.3.183.195:42811/ using document root <none> and password file <none>
I20260929 17:50:44.158874 19566 fs_manager.cc:359] Metadata directory not provided
I20260929 17:50:44.158946 19566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:50:44.159142 19566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:50:44.161305 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "ec11cab700dc4e83a2d4a9c8d499aea0"
format_stamp: "Formatted at 2026-09-29 17:50:44 on dist-test-slave-wbpt"
server_key: "c1105037b7fc304a3a92c25f22fbddeb"
server_key_iv: "c08967eb649b5148c31d740df8c97e93"
server_key_version: "encryptionkey@0"
I20260929 17:50:44.161851 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "ec11cab700dc4e83a2d4a9c8d499aea0"
format_stamp: "Formatted at 2026-09-29 17:50:44 on dist-test-slave-wbpt"
server_key: "c1105037b7fc304a3a92c25f22fbddeb"
server_key_iv: "c08967eb649b5148c31d740df8c97e93"
server_key_version: "encryptionkey@0"
I20260929 17:50:44.165750 19566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 17:50:44.168344 19581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:44.169684 19566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:50:44.169804 19566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "ec11cab700dc4e83a2d4a9c8d499aea0"
format_stamp: "Formatted at 2026-09-29 17:50:44 on dist-test-slave-wbpt"
server_key: "c1105037b7fc304a3a92c25f22fbddeb"
server_key_iv: "c08967eb649b5148c31d740df8c97e93"
server_key_version: "encryptionkey@0"
I20260929 17:50:44.169938 19566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:50:44.186791 19566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:50:44.190132 19566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:50:44.190335 19566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:50:44.190987 19566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:50:44.191870 19566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:50:44.191918 19566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:44.191972 19566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:50:44.192001 19566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:50:44.200688 19566 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:34301
I20260929 17:50:44.200716 19694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:34301 every 8 connection(s)
I20260929 17:50:44.201586 19566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:50:44.204895 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 19566
I20260929 17:50:44.205008 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:50:44.205261 3807 external_mini_cluster.cc:1428] Setting key eb3a7a1d9dd61a6010b8e87508d1f7c1
Sep 29 17:50:44 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704244, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:50:44.214766 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.203442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:47789 (local address 127.3.183.254:44201)
0929 17:50:44.203608 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.203612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.204602 (+ 990us) server_negotiation.cc:407] Connection header received
0929 17:50:44.205598 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.205602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.205650 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.205718 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:44.207129 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.207656 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.208895 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.209120 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.209193 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.211646 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:44.211649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:44.211651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:44.211676 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:44.213041 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:44.213498 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:44.213501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:44.213503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:44.213578 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:44.213831 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:44.213834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:44.213835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:44.214028 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:44.214100 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.214543 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.214646 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 17:50:44.215667 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.203757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:47789)
0929 17:50:44.204436 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.204477 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:44.205349 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:44.206047 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:44.206056 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:44.206506 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:50:44.206996 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.207011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.207819 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.207823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.208783 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.208794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.209248 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.209252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.209434 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.209954 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:50:44.209972 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:50:44.211491 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:50:44.213133 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:44.213137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:44.213149 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:44.213416 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:44.213643 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:50:44.213646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:50:44.213647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:50:44.213767 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:50:44.214108 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:50:44.214113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:50:44.214334 (+ 221us) client_negotiation.cc:769] Sending connection context
0929 17:50:44.214518 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 17:50:44.214773 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":113,"threads_started":1}
I20260929 17:50:44.216900 19695 heartbeater.cc:347] Connected to a master server at 127.3.183.254:44201
I20260929 17:50:44.217144 19695 heartbeater.cc:464] Registering TS with master...
I20260929 17:50:44.217593 19695 heartbeater.cc:511] Master 127.3.183.254:44201 requested a full tablet report, sending...
I20260929 17:50:44.218605 19236 ts_manager.cc:194] Registered new tserver with Master: ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301)
I20260929 17:50:44.219118 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:50:44.219267 19236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:47789
I20260929 17:50:44.228060 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.221200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.3.183.254:44201)
0929 17:50:44.221311 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.221315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.221631 (+ 316us) server_negotiation.cc:407] Connection header received
0929 17:50:44.221724 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.221726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.221766 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.221831 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:50:44.222515 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.223063 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.223858 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.224076 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.224162 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.225399 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:50:44.225402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:50:44.225404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:50:44.225426 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:50:44.226599 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:44.226938 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:44.226940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:44.226941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:44.227005 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:50:44.227245 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:50:44.227248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:50:44.227249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:50:44.227505 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:50:44.227620 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.227799 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.227938 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:50:44.230793 19236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:50:44.233875 19236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:50:44.247871 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.242765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34301 (local address 127.0.0.1:36450)
0929 17:50:44.243183 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.243194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:44.243256 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:44.244101 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:44.244104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:44.244120 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:44.244214 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.244222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.245232 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.245236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.246283 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.246293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.247307 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.247310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.247423 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.247461 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 17:50:44.247564 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 17:50:44.247685 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260929 17:50:44.247880 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.243176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.3.183.195:34301)
0929 17:50:44.243497 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.243505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.243523 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:50:44.243581 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.243584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.243799 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.243939 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:44.244330 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.245098 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.246402 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.247199 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.247291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.247452 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.247604 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.247696 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":130,"threads_started":1}
I20260929 17:50:44.248260 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.242081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45043 (local address 127.0.0.1:54812)
0929 17:50:44.242438 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.242497 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:44.242616 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:44.243207 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:44.243214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:44.243239 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:44.243394 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.243402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.244966 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.244970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.246319 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.246330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.247699 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.247703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.247825 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.247849 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:50:44.247907 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 17:50:44.248127 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":42,"thread_start_us":65,"threads_started":1}
I20260929 17:50:44.248383 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.242133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:36501 (local address 127.0.0.1:57818)
0929 17:50:44.243007 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.243020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:44.243104 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:44.244268 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:44.244271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:44.244288 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:44.244368 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.244374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.245776 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.245780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.246816 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.246825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.247871 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.247874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.247957 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.247976 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:50:44.248224 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 17:50:44.248286 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":102,"threads_started":1}
I20260929 17:50:44.248672 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.242208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.3.183.194:45043)
0929 17:50:44.242655 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.242668 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.242692 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:50:44.242771 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.242777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.243012 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.243557 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:44.243720 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.244825 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.246599 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.247399 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.247499 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.247635 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.248430 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.248531 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":126,"threads_started":1}
I20260929 17:50:44.249033 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.242833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.3.183.193:36501)
0929 17:50:44.243812 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.243826 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.243847 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:50:44.243918 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.243923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.244111 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.244518 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:44.244668 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.245640 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.246938 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.247720 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.248631 (+ 911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.248771 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.248818 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.248903 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":94,"threads_started":1}
I20260929 17:50:44.251292 19361 tablet_service.cc:1467] Processing CreateTablet for tablet 751cc190c8d947c2b528b9454436eca1 (DEFAULT_TABLE table=test-table [id=28176435ceac4982a0ae27363135d3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:44.251123 19629 tablet_service.cc:1467] Processing CreateTablet for tablet 751cc190c8d947c2b528b9454436eca1 (DEFAULT_TABLE table=test-table [id=28176435ceac4982a0ae27363135d3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:44.251307 19495 tablet_service.cc:1467] Processing CreateTablet for tablet 751cc190c8d947c2b528b9454436eca1 (DEFAULT_TABLE table=test-table [id=28176435ceac4982a0ae27363135d3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:50:44.252245 19361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751cc190c8d947c2b528b9454436eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:44.252256 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751cc190c8d947c2b528b9454436eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:44.252247 19495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751cc190c8d947c2b528b9454436eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:50:44.258293 19712 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Bootstrap starting.
I20260929 17:50:44.260125 19713 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Bootstrap starting.
I20260929 17:50:44.260125 19714 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Bootstrap starting.
I20260929 17:50:44.260805 19712 tablet_bootstrap.cc:654] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:44.261811 19712 log.cc:824] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:44.262699 19714 tablet_bootstrap.cc:654] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:44.262699 19713 tablet_bootstrap.cc:654] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Neither blocks nor log segments found. Creating new log.
I20260929 17:50:44.263597 19712 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: No bootstrap required, opened a new log
I20260929 17:50:44.263585 19713 log.cc:824] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:44.263585 19714 log.cc:824] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Log is configured to *not* fsync() on all Append() calls
I20260929 17:50:44.263819 19712 ts_tablet_manager.cc:1397] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:50:44.265707 19713 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: No bootstrap required, opened a new log
I20260929 17:50:44.265902 19713 ts_tablet_manager.cc:1397] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:50:44.266407 19714 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: No bootstrap required, opened a new log
I20260929 17:50:44.266585 19714 ts_tablet_manager.cc:1397] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 17:50:44.267318 19712 raft_consensus.cc:348] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.267539 19712 raft_consensus.cc:374] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:44.267598 19712 raft_consensus.cc:729] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a405a510e8ff49e7a5e7089746685b13, State: Initialized, Role: FOLLOWER
I20260929 17:50:44.268036 19712 consensus_queue.cc:260] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.268676 19427 heartbeater.cc:503] Master 127.3.183.254:44201 was elected leader, sending a full tablet report...
I20260929 17:50:44.268952 19712 ts_tablet_manager.cc:1428] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260929 17:50:44.270213 19714 raft_consensus.cc:348] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.270203 19713 raft_consensus.cc:348] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.270437 19714 raft_consensus.cc:374] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:44.270453 19713 raft_consensus.cc:374] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:50:44.270489 19714 raft_consensus.cc:729] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8499e8b033894eedbc118a4dd0072142, State: Initialized, Role: FOLLOWER
I20260929 17:50:44.270496 19713 raft_consensus.cc:729] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11cab700dc4e83a2d4a9c8d499aea0, State: Initialized, Role: FOLLOWER
I20260929 17:50:44.270937 19713 consensus_queue.cc:260] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.270947 19714 consensus_queue.cc:260] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.272004 19714 ts_tablet_manager.cc:1428] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:50:44.272008 19713 ts_tablet_manager.cc:1428] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260929 17:50:44.272327 19695 heartbeater.cc:503] Master 127.3.183.254:44201 was elected leader, sending a full tablet report...
I20260929 17:50:44.272436 19561 heartbeater.cc:503] Master 127.3.183.254:44201 was elected leader, sending a full tablet report...
I20260929 17:50:44.407312 19719 raft_consensus.cc:482] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:50:44.407529 19719 raft_consensus.cc:504] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.408630 19719 leader_election.cc:288] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a405a510e8ff49e7a5e7089746685b13 (127.3.183.193:36501), ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301)
I20260929 17:50:44.412127 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.408878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:36501 (local address 127.3.183.194:57353)
0929 17:50:44.409033 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.409049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:44.409147 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:44.409404 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:44.409407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:44.409425 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:44.409556 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.409561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.410251 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.410254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.411069 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.411078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.411689 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.411693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.411826 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.411846 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:50:44.411909 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:50:44.411983 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":3,"spinlock_wait_cycles":25856}
I20260929 17:50:44.412122 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.408976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:57353 (local address 127.3.183.193:36501)
0929 17:50:44.409125 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.409130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.409147 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:50:44.409244 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.409247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.409300 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.409383 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:44.409659 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.410151 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.411196 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.411583 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.411689 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.411743 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.411934 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.412015 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:50:44.412428 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.408879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34301 (local address 127.3.183.194:35361)
0929 17:50:44.409376 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.409386 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:50:44.409483 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:50:44.409700 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:50:44.409702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:50:44.409716 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:50:44.409820 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.409825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.410526 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.410530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.411329 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:50:44.411337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.411979 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.411982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:50:44.412074 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.412091 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:50:44.412206 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 17:50:44.412283 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":186,"spinlock_wait_cycles":384,"thread_start_us":130,"threads_started":1}
I20260929 17:50:44.412777 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.409015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:35361 (local address 127.3.183.195:34301)
0929 17:50:44.409128 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.409133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.409463 (+ 330us) server_negotiation.cc:407] Connection header received
0929 17:50:44.409576 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.409579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.409618 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.409675 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:50:44.409924 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.410422 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.411446 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.411873 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.411975 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.412025 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.412589 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.412663 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:50:44.412792 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751cc190c8d947c2b528b9454436eca1" candidate_uuid: "8499e8b033894eedbc118a4dd0072142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405a510e8ff49e7a5e7089746685b13" is_pre_election: true
I20260929 17:50:44.413105 19381 raft_consensus.cc:2454] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8499e8b033894eedbc118a4dd0072142 in term 0.
I20260929 17:50:44.413328 19649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751cc190c8d947c2b528b9454436eca1" candidate_uuid: "8499e8b033894eedbc118a4dd0072142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" is_pre_election: true
I20260929 17:50:44.413607 19649 raft_consensus.cc:2454] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8499e8b033894eedbc118a4dd0072142 in term 0.
I20260929 17:50:44.413602 19448 leader_election.cc:302] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8499e8b033894eedbc118a4dd0072142, a405a510e8ff49e7a5e7089746685b13; no voters:
I20260929 17:50:44.413858 19719 raft_consensus.cc:2781] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:50:44.413952 19719 raft_consensus.cc:482] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:50:44.414004 19719 raft_consensus.cc:3037] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:44.415047 19719 raft_consensus.cc:504] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.415361 19719 leader_election.cc:288] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [CANDIDATE]: Term 1 election: Requested vote from peers a405a510e8ff49e7a5e7089746685b13 (127.3.183.193:36501), ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301)
I20260929 17:50:44.415666 19649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751cc190c8d947c2b528b9454436eca1" candidate_uuid: "8499e8b033894eedbc118a4dd0072142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0"
I20260929 17:50:44.415695 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751cc190c8d947c2b528b9454436eca1" candidate_uuid: "8499e8b033894eedbc118a4dd0072142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405a510e8ff49e7a5e7089746685b13"
I20260929 17:50:44.415810 19649 raft_consensus.cc:3037] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:44.415844 19381 raft_consensus.cc:3037] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:50:44.417022 19649 raft_consensus.cc:2454] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8499e8b033894eedbc118a4dd0072142 in term 1.
I20260929 17:50:44.417022 19381 raft_consensus.cc:2454] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8499e8b033894eedbc118a4dd0072142 in term 1.
I20260929 17:50:44.417321 19448 leader_election.cc:302] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8499e8b033894eedbc118a4dd0072142, a405a510e8ff49e7a5e7089746685b13; no voters:
I20260929 17:50:44.417523 19719 raft_consensus.cc:2781] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:50:44.417718 19719 raft_consensus.cc:686] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 LEADER]: Becoming Leader. State: Replica: 8499e8b033894eedbc118a4dd0072142, State: Running, Role: LEADER
I20260929 17:50:44.418010 19719 consensus_queue.cc:237] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:50:44.421183 19234 catalog_manager.cc:5495] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8499e8b033894eedbc118a4dd0072142 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "8499e8b033894eedbc118a4dd0072142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } health_report { overall_health: UNKNOWN } } }
I20260929 17:50:44.444355 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.441072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.3.183.194:45043)
0929 17:50:44.441194 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:50:44.441198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:50:44.441211 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:50:44.441297 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:50:44.441299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:50:44.441351 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:50:44.441438 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:50:44.441693 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.442229 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.443077 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:50:44.443279 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:50:44.443345 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:50:44.443586 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:50:44.444027 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:50:44.444113 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:50:44.444199 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 17:50:44.444249 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:50:44.450073 19649 raft_consensus.cc:1261] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Refusing update from remote peer 8499e8b033894eedbc118a4dd0072142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:44.450073 19381 raft_consensus.cc:1261] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Refusing update from remote peer 8499e8b033894eedbc118a4dd0072142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:50:44.450696 19719 consensus_queue.cc:1035] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:50:44.450881 19722 consensus_queue.cc:1035] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 17:50:44.452459 19696 tablet.cc:2369] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:50:44.458469 19727 mvcc.cc:205] Tried to move back new op lower bound from 7334724585260589056 to 7334724585138638848. Current Snapshot: MvccSnapshot[applied={T|T < 7334724585260589056}]
I20260929 17:50:44.466677 19234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:44.466833 19234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:50:44.469401 19234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Sending DeleteTablet for 3 replicas of tablet 751cc190c8d947c2b528b9454436eca1
I20260929 17:50:44.470278 19495 tablet_service.cc:1514] Processing DeleteTablet for tablet 751cc190c8d947c2b528b9454436eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:44 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:54812
I20260929 17:50:44.470278 19629 tablet_service.cc:1514] Processing DeleteTablet for tablet 751cc190c8d947c2b528b9454436eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:44 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:36450
I20260929 17:50:44.470811 19361 tablet_service.cc:1514] Processing DeleteTablet for tablet 751cc190c8d947c2b528b9454436eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:44 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:57818
I20260929 17:50:44.470854 19735 tablet_replica.cc:330] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: stopping tablet replica
I20260929 17:50:44.471225 19735 raft_consensus.cc:2229] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:50:44.471240 19736 tablet_replica.cc:330] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: stopping tablet replica
I20260929 17:50:44.471298 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 19298
I20260929 17:50:44.471592 19735 raft_consensus.cc:2258] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:50:44.473330 19735 ts_tablet_manager.cc:1905] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:50:44.476261 19735 ts_tablet_manager.cc:1918] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:50:44.476365 19735 log.cc:1196] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/751cc190c8d947c2b528b9454436eca1
I20260929 17:50:44.476709 19735 ts_tablet_manager.cc:1939] T 751cc190c8d947c2b528b9454436eca1 P ec11cab700dc4e83a2d4a9c8d499aea0: Deleting consensus metadata
W20260929 17:50:44.477975 19220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:36501 (error 108)
I20260929 17:50:44.478111 19734 tablet_replica.cc:330] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: stopping tablet replica
I20260929 17:50:44.478199 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 19432
I20260929 17:50:44.478422 19734 raft_consensus.cc:2229] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:50:44.478713 19222 catalog_manager.cc:4841] TS ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301): tablet 751cc190c8d947c2b528b9454436eca1 (table test-table [id=28176435ceac4982a0ae27363135d3fe]) successfully deleted
I20260929 17:50:44.479656 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.479350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:36501 (local address 127.0.0.1:57826)
0929 17:50:44.479492 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.479566 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 17:50:44.479902 19220 catalog_manager.cc:4584] TS a405a510e8ff49e7a5e7089746685b13 (127.3.183.193:36501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 751cc190c8d947c2b528b9454436eca1: Network error: Client connection negotiation failed: client connection to 127.3.183.193:36501: connect: Connection refused (error 111)
I20260929 17:50:44.483803 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 19566
I20260929 17:50:44.485208 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:50:44.483973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45043 (local address 127.0.0.1:54830)
0929 17:50:44.485065 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0929 17:50:44.485124 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:45043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1007}
I20260929 17:50:44.488422 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 19204
I20260929 17:51:04.494052 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44201
--webserver_interface=127.3.183.254
--webserver_port=38655
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:04.602659 19738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:04.602967 19738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:04.603026 19738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:04.607189 19738 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 17:51:04.607254 19738 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 17:51:04.607275 19738 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 17:51:04.607293 19738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:04.607318 19738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:04.607348 19738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:04.607378 19738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:04.607402 19738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:04.612862 19738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:04.613940 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:04.614965 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:04.627910 19747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:04.627981 19738 server_base.cc:1034] running on GCE node
W20260929 17:51:04.627928 19745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:04.627936 19744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:04.628782 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:04.629925 19738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:04.631090 19738 hybrid_clock.cc:648] HybridClock initialized: now 1790704264631070 us; error 32 us; skew 500 ppm
Sep 29 17:51:04 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:04.638468 19738 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:51:04.639387 19738 webserver.cc:460] Webserver started at http://127.3.183.254:38655/ using document root <none> and password file <none>
I20260929 17:51:04.639976 19738 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:04.640053 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:04.643870 19738 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:51:04.645941 19754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:04.647219 19738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:04.647320 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "4315ebb0b43a41309f23145e433fbaac"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "f02251f69bb3b250dfb51a742e2e392d"
server_key_iv: "02d078496d31289aa1583621a7610004"
server_key_version: "encryptionkey@0"
I20260929 17:51:04.647948 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:04.679263 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:04.682837 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:04.683033 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:04.689947 19738 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:44201
I20260929 17:51:04.690191 19806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:44201 every 8 connection(s)
I20260929 17:51:04.691064 19738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:04.695456 19807 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:51:04.697194 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Bootstrap starting.
I20260929 17:51:04.701010 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 19738
I20260929 17:51:04.701495 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:36501
--local_ip_for_outbound_sockets=127.3.183.193
--tserver_master_addrs=127.3.183.254:44201
--webserver_port=39753
--webserver_interface=127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:51:04.715588 19807 log.cc:824] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:04.725529 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:51:04.725906 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Bootstrap complete.
I20260929 17:51:04.729923 19807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:04.730327 19807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4315ebb0b43a41309f23145e433fbaac, State: Initialized, Role: FOLLOWER
I20260929 17:51:04.730822 19807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:04.730962 19807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:04.731038 19807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:04.731132 19807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:51:04.733085 19807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:04.733407 19807 leader_election.cc:302] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4315ebb0b43a41309f23145e433fbaac; no voters:
I20260929 17:51:04.733711 19807 leader_election.cc:288] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:51:04.733817 19811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:51:04.734021 19811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 LEADER]: Becoming Leader. State: Replica: 4315ebb0b43a41309f23145e433fbaac, State: Running, Role: LEADER
I20260929 17:51:04.734397 19811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:04.735018 19807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:04.735738 19813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4315ebb0b43a41309f23145e433fbaac. Latest consensus state: current_term: 2 leader_uuid: "4315ebb0b43a41309f23145e433fbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } } }
I20260929 17:51:04.736070 19813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:04.736429 19820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:04.735929 19812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4315ebb0b43a41309f23145e433fbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } } }
I20260929 17:51:04.736613 19812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:04.741297 19820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:04.742015 19820 catalog_manager.cc:1259] Loaded cluster ID: b7c29504a15f47329e7551ebf6c58197
I20260929 17:51:04.742076 19820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:04.743984 19820 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:04.744678 19820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Loaded TSK: 0
I20260929 17:51:04.753777 19820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Generated new TSK 1
I20260929 17:51:04.754047 19820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:51:04.841821 19809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:04.842096 19809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:04.842140 19809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:04.846181 19809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:04.846244 19809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:04.846306 19809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:04.851662 19809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:36501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=39753
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:04.852690 19809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:04.853688 19809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:04.862785 19834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:04.862803 19835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:04.863056 19809 server_base.cc:1034] running on GCE node
W20260929 17:51:04.863154 19837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:04.863605 19809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:04.864243 19809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:04.865401 19809 hybrid_clock.cc:648] HybridClock initialized: now 1790704264865376 us; error 45 us; skew 500 ppm
Sep 29 17:51:04 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:04.868012 19809 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:51:04.868860 19809 webserver.cc:460] Webserver started at http://127.3.183.193:39753/ using document root <none> and password file <none>
I20260929 17:51:04.869379 19809 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:04.869457 19809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:04.873203 19809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:51:04.875356 19844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:04.876734 19809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:04.876844 19809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "a405a510e8ff49e7a5e7089746685b13"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "bb7b62f64061b7341d74f2053448e78a"
server_key_iv: "95c4d8eb9b49b33f7cd085b488364adb"
server_key_version: "encryptionkey@0"
I20260929 17:51:04.877394 19809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:04.891762 19809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:04.895048 19809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:04.895249 19809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:04.895918 19809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:04.897049 19851 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:51:04.901156 19809 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:51:04.901217 19809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 17:51:04.901268 19809 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:51:04.902962 19809 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:51:04.903020 19809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:51:04.903290 19851 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Bootstrap starting.
I20260929 17:51:04.913549 19809 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:36501
I20260929 17:51:04.913559 19958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:36501 every 8 connection(s)
I20260929 17:51:04.914808 19809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:04.918568 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 19809
I20260929 17:51:04.919034 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:45043
--local_ip_for_outbound_sockets=127.3.183.194
--tserver_master_addrs=127.3.183.254:44201
--webserver_port=34983
--webserver_interface=127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:51:04.924808 19851 log.cc:824] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:51:04 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:04.932549 19851 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:51:04.933035 19851 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Bootstrap complete.
I20260929 17:51:04.933342 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:04.916940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:36983 (local address 127.3.183.254:44201)
0929 17:51:04.917765 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:04.917779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:04.917816 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:51:04.919067 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:04.919083 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:04.919489 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:04.919868 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:04.921406 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.922418 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.924001 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.924721 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.924901 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:04.927858 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:04.927867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:04.927885 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:04.927923 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:04.929957 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:04.930553 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:04.930560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:04.930566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:04.930721 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:04.931073 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:04.931076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:04.931077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:04.931503 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:04.931628 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:04.932244 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 17:51:04.932534 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":120,"threads_started":1}
I20260929 17:51:04.933383 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:04.916801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:36983)
0929 17:51:04.917374 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:04.917447 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:04.918548 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:04.920327 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:04.920335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:04.920784 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:04.921239 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:04.921252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.922567 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.922571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:04.923873 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:04.923886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.924880 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.924885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:04.925103 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:04.925767 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:04.925789 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:04.927614 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:04.930084 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:04.930090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:04.930104 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:04.930433 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:04.930821 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:04.930824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:04.930826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:04.930985 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:04.931615 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:04.931621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:04.931925 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 17:51:04.932110 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 17:51:04.932441 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260929 17:51:04.934269 19851 ts_tablet_manager.cc:1397] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260929 17:51:04.936180 19959 heartbeater.cc:347] Connected to a master server at 127.3.183.254:44201
I20260929 17:51:04.936501 19959 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:04.937038 19959 heartbeater.cc:511] Master 127.3.183.254:44201 requested a full tablet report, sending...
I20260929 17:51:04.937206 19851 raft_consensus.cc:348] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:51:04.937659 19851 raft_consensus.cc:729] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a405a510e8ff49e7a5e7089746685b13, State: Initialized, Role: FOLLOWER
I20260929 17:51:04.938071 19851 consensus_queue.cc:260] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:51:04.938951 19851 ts_tablet_manager.cc:1428] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:51:04.941094 19771 ts_manager.cc:194] Registered new tserver with Master: a405a510e8ff49e7a5e7089746685b13 (127.3.183.193:36501)
I20260929 17:51:04.945181 19771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:36983
I20260929 17:51:04.948083 19959 heartbeater.cc:503] Master 127.3.183.254:44201 was elected leader, sending a full tablet report...
Sep 29 17:51:04 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.193@KRBTEST.COM
I20260929 17:51:04.954912 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:04.944460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:36501 (local address 127.0.0.1:45668)
0929 17:51:04.945200 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:04.945233 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:04.945376 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:04.945873 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:04.945879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:04.946228 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:04.946348 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:04.946355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.947319 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.947322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:04.947984 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:04.947994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.948778 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.948782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:04.948886 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:04.949481 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:04.949494 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:04.951044 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:04.953050 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:04.953058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:04.953064 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:04.953285 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:04.953743 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:04.953747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:04.953749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:04.953837 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:04.954270 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:04.954276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:04.954393 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 17:51:04.954509 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 17:51:04.954712 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":119,"threads_started":1}
I20260929 17:51:04.954912 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:04.944588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.3.183.193:36501)
0929 17:51:04.944972 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:04.944982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:04.945499 (+ 517us) server_negotiation.cc:407] Connection header received
0929 17:51:04.945563 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:04.945567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:04.945723 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:04.945835 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:04.946464 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.947211 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.948089 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:04.948572 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:04.948659 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:04.951221 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:04.951232 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:04.951245 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:04.951273 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:04.952947 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:04.953437 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:04.953443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:04.953445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:04.953626 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:04.953922 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:04.953925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:04.953928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:04.954162 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:04.954274 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:04.954610 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 17:51:04.954768 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":289,"thread_start_us":88,"threads_started":1}
I20260929 17:51:04.956007 19893 tablet_service.cc:1514] Processing DeleteTablet for tablet 751cc190c8d947c2b528b9454436eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:44 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45668
I20260929 17:51:04.957006 19968 tablet_replica.cc:330] stopping tablet replica
I20260929 17:51:04.957293 19968 raft_consensus.cc:2229] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:04.957526 19968 raft_consensus.cc:2258] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:04.958468 19968 ts_tablet_manager.cc:1905] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:04.961863 19968 ts_tablet_manager.cc:1918] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:04.961956 19968 log.cc:1196] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal/wals/751cc190c8d947c2b528b9454436eca1
I20260929 17:51:04.962169 19968 ts_tablet_manager.cc:1939] T 751cc190c8d947c2b528b9454436eca1 P a405a510e8ff49e7a5e7089746685b13: Deleting consensus metadata
I20260929 17:51:04.963927 19755 catalog_manager.cc:4841] TS a405a510e8ff49e7a5e7089746685b13 (127.3.183.193:36501): tablet 751cc190c8d947c2b528b9454436eca1 (table test-table [id=28176435ceac4982a0ae27363135d3fe]) successfully deleted
W20260929 17:51:05.037894 19963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:05.038156 19963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:05.038204 19963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:05.042243 19963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:05.042310 19963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:05.042387 19963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:05.047617 19963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:45043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=34983
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:05.048671 19963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:05.049667 19963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:05.059388 19973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:05.059374 19976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:05.059721 19974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:05.059713 19963 server_base.cc:1034] running on GCE node
I20260929 17:51:05.060150 19963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:05.060810 19963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:05.061986 19963 hybrid_clock.cc:648] HybridClock initialized: now 1790704265061950 us; error 45 us; skew 500 ppm
Sep 29 17:51:05 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:05.064633 19963 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:51:05.065490 19963 webserver.cc:460] Webserver started at http://127.3.183.194:34983/ using document root <none> and password file <none>
I20260929 17:51:05.066004 19963 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:05.066082 19963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:05.069875 19963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:05.071954 19983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:05.073166 19963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:05.073271 19963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "8499e8b033894eedbc118a4dd0072142"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "621621c4b56bf0a8841194c6d5ffc871"
server_key_iv: "85f8cda0d357a07810a47792fca24a40"
server_key_version: "encryptionkey@0"
I20260929 17:51:05.073837 19963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:05.088629 19963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:05.091557 19963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:05.091722 19963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:05.092290 19963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:05.093353 19990 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:51:05.097262 19963 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:51:05.097324 19963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 17:51:05.097395 19963 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:51:05.099109 19963 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:51:05.099154 19963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:51:05.099397 19990 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Bootstrap starting.
I20260929 17:51:05.108430 19963 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:45043
I20260929 17:51:05.108496 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:45043 every 8 connection(s)
I20260929 17:51:05.109337 19963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:05.116688 19990 log.cc:824] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:05.117018 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 19963
I20260929 17:51:05.117427 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:34301
--local_ip_for_outbound_sockets=127.3.183.195
--tserver_master_addrs=127.3.183.254:44201
--webserver_port=42811
--webserver_interface=127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:51:05 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:05.123265 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.111367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:55633)
0929 17:51:05.111928 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:05.111971 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:05.112778 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:05.113273 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:05.113280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:05.113626 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:05.114131 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.114146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.114924 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.114926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.115772 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.115839 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.116276 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.116280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.116453 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.116986 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:05.117002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:05.118523 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:05.120402 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:05.120408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:05.120421 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:05.120657 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:05.120960 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:05.120964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:05.120965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:05.121129 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:05.121540 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:05.121546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:05.121827 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 17:51:05.122011 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 17:51:05.122318 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":140,"threads_started":1}
I20260929 17:51:05.124213 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.111167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:55633 (local address 127.3.183.254:44201)
0929 17:51:05.111292 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.111295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.112066 (+ 771us) server_negotiation.cc:407] Connection header received
0929 17:51:05.112983 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.112987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.113045 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.113113 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:05.114270 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.114791 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.115930 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.116161 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.116250 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.118678 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:05.118681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:05.118683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:05.118712 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:05.120302 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.120745 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.120749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.120750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.120837 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.121230 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.121233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.121234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.121443 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:05.121531 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.123912 (+ 2381us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.124052 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:51:05.124570 19990 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:51:05.124995 19990 tablet_bootstrap.cc:492] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Bootstrap complete.
I20260929 17:51:05.125191 20098 heartbeater.cc:347] Connected to a master server at 127.3.183.254:44201
I20260929 17:51:05.125454 20098 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:05.126008 19990 ts_tablet_manager.cc:1397] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.000s
I20260929 17:51:05.126024 20098 heartbeater.cc:511] Master 127.3.183.254:44201 requested a full tablet report, sending...
I20260929 17:51:05.128360 19771 ts_manager.cc:194] Registered new tserver with Master: 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
I20260929 17:51:05.128397 19990 raft_consensus.cc:348] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:51:05.128918 19990 raft_consensus.cc:729] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8499e8b033894eedbc118a4dd0072142, State: Initialized, Role: FOLLOWER
I20260929 17:51:05.129426 19990 consensus_queue.cc:260] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } }
I20260929 17:51:05.129670 19771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:55633
I20260929 17:51:05.130455 19990 ts_tablet_manager.cc:1428] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260929 17:51:05.131892 20098 heartbeater.cc:503] Master 127.3.183.254:44201 was elected leader, sending a full tablet report...
Sep 29 17:51:05 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.194@KRBTEST.COM
I20260929 17:51:05.139747 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.129974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.3.183.194:45043)
0929 17:51:05.130267 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.130278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.130299 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:51:05.130373 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.130378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.130631 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.131225 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:05.131338 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.132563 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.133540 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.134057 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.134151 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.136513 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:05.136523 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:05.136535 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:05.136556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:05.138083 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.138564 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.138570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.138572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.138653 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.138897 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.138899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.138900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.139122 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:05.139221 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.139505 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.139626 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260929 17:51:05.139744 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.129350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45043 (local address 127.0.0.1:33244)
0929 17:51:05.129619 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:05.129633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:05.129708 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:05.130772 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:05.130775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:05.130969 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:05.131077 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.131089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.132673 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.132677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.133440 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.133449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.134158 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.134161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.134228 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.134730 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:05.134743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:05.136277 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:05.138196 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:05.138199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:05.138202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:05.138457 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:05.138731 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:05.138733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:05.138735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:05.138820 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:05.139205 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:05.139208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:05.139308 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 17:51:05.139416 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 17:51:05.139564 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260929 17:51:05.140923 20032 tablet_service.cc:1514] Processing DeleteTablet for tablet 751cc190c8d947c2b528b9454436eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:50:44 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:33244
I20260929 17:51:05.141748 20105 tablet_replica.cc:330] stopping tablet replica
I20260929 17:51:05.142061 20105 raft_consensus.cc:2229] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:05.142313 20105 raft_consensus.cc:2258] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:05.143224 20105 ts_tablet_manager.cc:1905] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:05.146795 20105 ts_tablet_manager.cc:1918] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:05.146875 20105 log.cc:1196] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/751cc190c8d947c2b528b9454436eca1
I20260929 17:51:05.147107 20105 ts_tablet_manager.cc:1939] T 751cc190c8d947c2b528b9454436eca1 P 8499e8b033894eedbc118a4dd0072142: Deleting consensus metadata
I20260929 17:51:05.148880 19757 catalog_manager.cc:4841] TS 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043): tablet 751cc190c8d947c2b528b9454436eca1 (table test-table [id=28176435ceac4982a0ae27363135d3fe]) successfully deleted
W20260929 17:51:05.231844 20102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:05.232142 20102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:05.232204 20102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:05.236378 20102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:05.236466 20102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:05.236541 20102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:05.241896 20102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:34301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=42811
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:05.242923 20102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:05.243944 20102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:05.254046 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:05.254024 20113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:05.254069 20111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:05.254382 20102 server_base.cc:1034] running on GCE node
I20260929 17:51:05.254958 20102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:05.255694 20102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:05.256901 20102 hybrid_clock.cc:648] HybridClock initialized: now 1790704265256881 us; error 30 us; skew 500 ppm
Sep 29 17:51:05 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:05.259720 20102 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:51:05.260685 20102 webserver.cc:460] Webserver started at http://127.3.183.195:42811/ using document root <none> and password file <none>
I20260929 17:51:05.261225 20102 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:05.261294 20102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:05.265172 20102 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:51:05.267336 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:05.268857 20102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:51:05.268985 20102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "ec11cab700dc4e83a2d4a9c8d499aea0"
format_stamp: "Formatted at 2026-09-29 17:50:44 on dist-test-slave-wbpt"
server_key: "c1105037b7fc304a3a92c25f22fbddeb"
server_key_iv: "c08967eb649b5148c31d740df8c97e93"
server_key_version: "encryptionkey@0"
I20260929 17:51:05.269567 20102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:05.292557 20102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:05.295390 20102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:05.295558 20102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:05.296169 20102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:05.296977 20102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:05.297021 20102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:05.297062 20102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:05.297106 20102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:05.305469 20102 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:34301
I20260929 17:51:05.305516 20233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:34301 every 8 connection(s)
I20260929 17:51:05.306432 20102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:51:05 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:05.315265 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 20102
I20260929 17:51:05.319391 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.308196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:54809 (local address 127.3.183.254:44201)
0929 17:51:05.308354 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.308358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.309298 (+ 940us) server_negotiation.cc:407] Connection header received
0929 17:51:05.310141 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.310144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.310189 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.310257 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:05.311389 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.311945 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.313059 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.313266 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.313330 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.315841 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:05.315845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:05.315848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:05.315875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:05.317444 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.318005 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.318007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.318008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.318075 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.318392 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.318395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.318396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.318601 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:05.318694 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.319125 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.319224 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:51:05.320569 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.308560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:54809)
0929 17:51:05.309179 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:05.309218 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:05.309972 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:05.310434 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:05.310440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:05.310786 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:05.311255 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.311273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.312064 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.312067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.312961 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.312971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.313367 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.313370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.313506 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.314083 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:05.314099 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:05.315617 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:05.317570 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:05.317576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:05.317591 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:05.317912 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:05.318144 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:05.318147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:05.318149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:05.318311 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:05.318685 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:05.318690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:05.318908 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 17:51:05.319065 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 17:51:05.319436 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":114,"threads_started":1}
I20260929 17:51:05.321673 20234 heartbeater.cc:347] Connected to a master server at 127.3.183.254:44201
I20260929 17:51:05.321933 20234 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:05.322405 20234 heartbeater.cc:511] Master 127.3.183.254:44201 requested a full tablet report, sending...
I20260929 17:51:05.323441 19771 ts_manager.cc:194] Registered new tserver with Master: ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301)
I20260929 17:51:05.324133 19771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:54809
I20260929 17:51:05.324285 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.315912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.3.183.254:44201)
0929 17:51:05.316231 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.316236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.316312 (+ 76us) server_negotiation.cc:407] Connection header received
0929 17:51:05.316420 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.316423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.316468 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.316536 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:05.317394 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.318145 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.319092 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.319434 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.319529 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.321167 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:05.321171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:05.321173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:05.321195 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:05.322642 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.323090 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.323093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.323095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.323176 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.323415 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.323418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.323420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.323629 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:05.323835 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.323973 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.324091 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":96,"threads_started":1}
I20260929 17:51:05.325789 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:51:05.329564 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.326320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.3.183.254:44201)
0929 17:51:05.326423 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.326427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.326594 (+ 167us) server_negotiation.cc:407] Connection header received
0929 17:51:05.326737 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.326740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.326823 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.326902 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:05.327126 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.327608 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.328476 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.328710 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.328791 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.329102 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:05.329220 (+ 118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 17:51:05.329439 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49792: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 17:51:05.329617 20237 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49792: token expired
W20260929 17:51:05.329876 19701 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.183.254:44201)
I20260929 17:51:05.336194 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.330182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.3.183.254:44201)
0929 17:51:05.330287 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.330290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.330303 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:51:05.330387 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.330389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.330420 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.330476 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:05.330951 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.331399 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.332168 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.332363 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.332428 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.333540 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:05.333543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:05.333545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:05.333559 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:05.334797 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.335172 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.335175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.335177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.335251 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:05.335503 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:05.335505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:05.335507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:05.335651 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:05.335742 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.335917 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.336006 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:51:05.338490 19771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:51:05.341718 19771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:51:05.348795 19893 tablet_service.cc:1467] Processing CreateTablet for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 (DEFAULT_TABLE table=test-table [id=a12df840685e471c8754724df79a26a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:05.348795 20032 tablet_service.cc:1467] Processing CreateTablet for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 (DEFAULT_TABLE table=test-table [id=a12df840685e471c8754724df79a26a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:05.349422 20032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:05.349422 19893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:05.352200 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.347823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34301 (local address 127.0.0.1:59726)
0929 17:51:05.347944 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:05.347959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:05.348043 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:05.348897 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:05.348900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:05.348918 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:05.349021 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.349028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.350078 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.350082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.350909 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.350920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.351766 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.351769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.351879 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.351921 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 17:51:05.351987 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:51:05.352066 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 17:51:05.352263 19990 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Bootstrap starting.
I20260929 17:51:05.352413 19851 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Bootstrap starting.
I20260929 17:51:05.353235 19851 tablet_bootstrap.cc:654] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:05.353366 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.348164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.3.183.195:34301)
0929 17:51:05.348493 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.348502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.348522 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:51:05.348591 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.348595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.348764 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.348907 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:05.349140 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.349930 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.351033 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.351656 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.351753 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.352019 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.352080 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.353225 (+ 1145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":156,"threads_started":1}
I20260929 17:51:05.353404 19990 tablet_bootstrap.cc:654] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:05.354753 19851 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: No bootstrap required, opened a new log
I20260929 17:51:05.354851 19851 ts_tablet_manager.cc:1397] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:05.355288 19851 raft_consensus.cc:348] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.355414 19851 raft_consensus.cc:374] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:05.355461 19851 raft_consensus.cc:729] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a405a510e8ff49e7a5e7089746685b13, State: Initialized, Role: FOLLOWER
I20260929 17:51:05.355597 19851 consensus_queue.cc:260] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.355886 19851 ts_tablet_manager.cc:1428] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:51:05.356397 19990 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: No bootstrap required, opened a new log
I20260929 17:51:05.356489 19990 ts_tablet_manager.cc:1397] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:51:05.356922 19990 raft_consensus.cc:348] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.357038 19990 raft_consensus.cc:374] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:05.357095 19990 raft_consensus.cc:729] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8499e8b033894eedbc118a4dd0072142, State: Initialized, Role: FOLLOWER
I20260929 17:51:05.356664 20168 tablet_service.cc:1467] Processing CreateTablet for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 (DEFAULT_TABLE table=test-table [id=a12df840685e471c8754724df79a26a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:05.357228 19990 consensus_queue.cc:260] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.357508 19990 ts_tablet_manager.cc:1428] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:51:05.357829 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 17:51:05.360493 20099 tablet.cc:2369] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:51:05.365260 20246 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Bootstrap starting.
I20260929 17:51:05.367494 20246 tablet_bootstrap.cc:654] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:05.368355 20246 log.cc:824] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:05.370098 20246 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: No bootstrap required, opened a new log
I20260929 17:51:05.370280 20246 ts_tablet_manager.cc:1397] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:51:05.373318 20246 raft_consensus.cc:348] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.373553 20246 raft_consensus.cc:374] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:05.373616 20246 raft_consensus.cc:729] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec11cab700dc4e83a2d4a9c8d499aea0, State: Initialized, Role: FOLLOWER
I20260929 17:51:05.374032 20246 consensus_queue.cc:260] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.374648 20234 heartbeater.cc:503] Master 127.3.183.254:44201 was elected leader, sending a full tablet report...
I20260929 17:51:05.374979 20246 ts_tablet_manager.cc:1428] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 17:51:05.416163 19960 tablet.cc:2369] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:51:05.463169 19965 raft_consensus.cc:482] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:05.463378 19965 raft_consensus.cc:504] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.464533 19965 leader_election.cc:288] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301), 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
I20260929 17:51:05.468391 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.464884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:40895 (local address 127.3.183.195:34301)
0929 17:51:05.465004 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.465008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.465088 (+ 80us) server_negotiation.cc:407] Connection header received
0929 17:51:05.465312 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.465316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.465367 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.465416 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:05.465800 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.466346 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.467407 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.467851 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.467942 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.467996 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.468216 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.468279 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:51:05.468392 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.465233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:51951 (local address 127.3.183.194:45043)
0929 17:51:05.465385 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.465389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.465404 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:51:05.465517 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.465520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.465591 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.465678 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:05.465918 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.466408 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.467417 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.467890 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.467994 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.468137 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.468209 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.468274 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:51:05.468410 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.465128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45043 (local address 127.3.183.193:51951)
0929 17:51:05.465322 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:05.465335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:05.465421 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:05.465694 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:05.465696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:05.465712 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:05.465800 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.465806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.466520 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.466523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.467300 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.467309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.468017 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.468021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.468102 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.468119 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:51:05.468180 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:51:05.468247 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":22,"thread_start_us":57,"threads_started":1}
I20260929 17:51:05.468427 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.464775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34301 (local address 127.3.183.193:40895)
0929 17:51:05.465002 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:05.465016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:05.465145 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:05.465560 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:05.465563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:05.465582 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:05.465684 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.465691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.466449 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.466453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.467288 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:05.467297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.467952 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.467956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:05.468067 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.468110 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 17:51:05.468195 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:51:05.468267 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":29,"thread_start_us":69,"threads_started":1}
I20260929 17:51:05.469144 20052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e45e1db59fd4f9cb0b0f24d9082ccc9" candidate_uuid: "a405a510e8ff49e7a5e7089746685b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8499e8b033894eedbc118a4dd0072142" is_pre_election: true
I20260929 17:51:05.469184 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e45e1db59fd4f9cb0b0f24d9082ccc9" candidate_uuid: "a405a510e8ff49e7a5e7089746685b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" is_pre_election: true
I20260929 17:51:05.469456 20052 raft_consensus.cc:2454] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a405a510e8ff49e7a5e7089746685b13 in term 0.
I20260929 17:51:05.469456 20188 raft_consensus.cc:2454] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a405a510e8ff49e7a5e7089746685b13 in term 0.
I20260929 17:51:05.469874 19847 leader_election.cc:302] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a405a510e8ff49e7a5e7089746685b13, ec11cab700dc4e83a2d4a9c8d499aea0; no voters:
I20260929 17:51:05.470112 19965 raft_consensus.cc:2781] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:51:05.470196 19965 raft_consensus.cc:482] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:51:05.470245 19965 raft_consensus.cc:3037] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:05.471210 19965 raft_consensus.cc:504] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.471508 19965 leader_election.cc:288] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 1 election: Requested vote from peers ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301), 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
I20260929 17:51:05.471920 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e45e1db59fd4f9cb0b0f24d9082ccc9" candidate_uuid: "a405a510e8ff49e7a5e7089746685b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0"
I20260929 17:51:05.471921 20052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e45e1db59fd4f9cb0b0f24d9082ccc9" candidate_uuid: "a405a510e8ff49e7a5e7089746685b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8499e8b033894eedbc118a4dd0072142"
I20260929 17:51:05.472036 20188 raft_consensus.cc:3037] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:05.472055 20052 raft_consensus.cc:3037] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:05.473034 20052 raft_consensus.cc:2454] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a405a510e8ff49e7a5e7089746685b13 in term 1.
I20260929 17:51:05.473121 20188 raft_consensus.cc:2454] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a405a510e8ff49e7a5e7089746685b13 in term 1.
I20260929 17:51:05.473279 19847 leader_election.cc:302] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8499e8b033894eedbc118a4dd0072142, a405a510e8ff49e7a5e7089746685b13; no voters:
I20260929 17:51:05.473431 19965 raft_consensus.cc:2781] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:05.473608 19965 raft_consensus.cc:686] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 LEADER]: Becoming Leader. State: Replica: a405a510e8ff49e7a5e7089746685b13, State: Running, Role: LEADER
I20260929 17:51:05.473867 19965 consensus_queue.cc:237] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:05.476548 19771 catalog_manager.cc:5495] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to a405a510e8ff49e7a5e7089746685b13 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "a405a510e8ff49e7a5e7089746685b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } health_report { overall_health: UNKNOWN } } }
I20260929 17:51:05.513670 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.509889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.3.183.193:36501)
0929 17:51:05.510189 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:05.510198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:05.510211 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:51:05.510288 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:05.510290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:05.510356 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:05.510471 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:05.510750 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.511274 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.512192 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:05.512436 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:05.512536 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:05.512816 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:05.513241 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:51:05.513363 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:05.513438 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 17:51:05.513507 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
I20260929 17:51:05.517689 20052 raft_consensus.cc:1261] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Refusing update from remote peer a405a510e8ff49e7a5e7089746685b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:05.517694 20188 raft_consensus.cc:1261] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Refusing update from remote peer a405a510e8ff49e7a5e7089746685b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:05.518329 19965 consensus_queue.cc:1035] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:05.518528 20251 consensus_queue.cc:1035] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:05.524554 20256 mvcc.cc:205] Tried to move back new op lower bound from 7334724671554207744 to 7334724671383597056. Current Snapshot: MvccSnapshot[applied={T|T < 7334724671554207744}]
I20260929 17:51:05.530889 20257 mvcc.cc:205] Tried to move back new op lower bound from 7334724671554207744 to 7334724671383597056. Current Snapshot: MvccSnapshot[applied={T|T < 7334724671554207744}]
I20260929 17:51:05.531700 19769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:05.531881 19769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:05.534035 19769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Sending DeleteTablet for 3 replicas of tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9
I20260929 17:51:05.534709 19893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:05 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:45668
I20260929 17:51:05.534739 20032 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:05 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:33244
I20260929 17:51:05.534757 20168 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:05 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:59726
I20260929 17:51:05.534863 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 19809
I20260929 17:51:05.534968 20105 tablet_replica.cc:330] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: stopping tablet replica
I20260929 17:51:05.535089 20105 raft_consensus.cc:2229] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:05.535202 20105 raft_consensus.cc:2258] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:05.536028 20266 tablet_replica.cc:330] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: stopping tablet replica
I20260929 17:51:05.536298 20266 raft_consensus.cc:2229] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:05.536566 20266 raft_consensus.cc:2258] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:05.537603 20105 ts_tablet_manager.cc:1905] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:05.537963 20266 ts_tablet_manager.cc:1905] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:05.540071 20105 ts_tablet_manager.cc:1918] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:05.540153 20105 log.cc:1196] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/5e45e1db59fd4f9cb0b0f24d9082ccc9
I20260929 17:51:05.540392 20105 ts_tablet_manager.cc:1939] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P 8499e8b033894eedbc118a4dd0072142: Deleting consensus metadata
I20260929 17:51:05.540437 20266 ts_tablet_manager.cc:1918] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:05.540519 20266 log.cc:1196] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/5e45e1db59fd4f9cb0b0f24d9082ccc9
I20260929 17:51:05.540815 20266 ts_tablet_manager.cc:1939] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P ec11cab700dc4e83a2d4a9c8d499aea0: Deleting consensus metadata
I20260929 17:51:05.541278 19757 catalog_manager.cc:4841] TS 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043): tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 (table test-table [id=a12df840685e471c8754724df79a26a1]) successfully deleted
W20260929 17:51:05.541347 19755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:36501 (error 108)
I20260929 17:51:05.541970 19757 catalog_manager.cc:4841] TS ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301): tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9 (table test-table [id=a12df840685e471c8754724df79a26a1]) successfully deleted
I20260929 17:51:05.542326 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:05.542047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:36501 (local address 127.0.0.1:45686)
0929 17:51:05.542181 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:05.542247 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 17:51:05.542408 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 19963
W20260929 17:51:05.542479 19755 catalog_manager.cc:4584] TS a405a510e8ff49e7a5e7089746685b13 (127.3.183.193:36501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e45e1db59fd4f9cb0b0f24d9082ccc9: Network error: Client connection negotiation failed: client connection to 127.3.183.193:36501: connect: Connection refused (error 111)
I20260929 17:51:05.547392 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 20102
I20260929 17:51:05.552249 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 19738
I20260929 17:51:25.558022 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44201
--webserver_interface=127.3.183.254
--webserver_port=38655
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:25.665841 20268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:25.666139 20268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:25.666204 20268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:25.670246 20268 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 17:51:25.670323 20268 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 17:51:25.670351 20268 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 17:51:25.670372 20268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:25.670401 20268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:25.670440 20268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:25.670475 20268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:25.670517 20268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:25.676050 20268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:25.677259 20268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:25.678372 20268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:25.687335 20274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:25.687440 20268 server_base.cc:1034] running on GCE node
W20260929 17:51:25.687476 20276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:25.687461 20273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:25.688269 20268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:25.689416 20268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:25.690589 20268 hybrid_clock.cc:648] HybridClock initialized: now 1790704285690571 us; error 30 us; skew 500 ppm
Sep 29 17:51:25 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704285, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:25.697047 20268 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:51:25.697927 20268 webserver.cc:460] Webserver started at http://127.3.183.254:38655/ using document root <none> and password file <none>
I20260929 17:51:25.698444 20268 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:25.698503 20268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:25.702306 20268 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:25.704596 20283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:25.705958 20268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:25.706082 20268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "4315ebb0b43a41309f23145e433fbaac"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "f02251f69bb3b250dfb51a742e2e392d"
server_key_iv: "02d078496d31289aa1583621a7610004"
server_key_version: "encryptionkey@0"
I20260929 17:51:25.706625 20268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:25.728332 20268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:25.731477 20268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:25.731634 20268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:25.738426 20268 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:44201
I20260929 17:51:25.738677 20335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:44201 every 8 connection(s)
I20260929 17:51:25.739487 20268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:25.743636 20336 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:51:25.744427 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 20268
I20260929 17:51:25.744891 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:36501
--local_ip_for_outbound_sockets=127.3.183.193
--tserver_master_addrs=127.3.183.254:44201
--webserver_port=39753
--webserver_interface=127.3.183.193
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:51:25.745434 20336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Bootstrap starting.
I20260929 17:51:25.762844 20336 log.cc:824] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:25.773823 20336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:51:25.774199 20336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: Bootstrap complete.
I20260929 17:51:25.778101 20336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:25.778509 20336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4315ebb0b43a41309f23145e433fbaac, State: Initialized, Role: FOLLOWER
I20260929 17:51:25.779024 20336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:25.779160 20336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:25.779228 20336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:25.779325 20336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:51:25.781235 20336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:25.781545 20336 leader_election.cc:302] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4315ebb0b43a41309f23145e433fbaac; no voters:
I20260929 17:51:25.781831 20336 leader_election.cc:288] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 17:51:25.781973 20340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 3 FOLLOWER]: Leader election won for term 3
I20260929 17:51:25.782222 20340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [term 3 LEADER]: Becoming Leader. State: Replica: 4315ebb0b43a41309f23145e433fbaac, State: Running, Role: LEADER
I20260929 17:51:25.782603 20340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } }
I20260929 17:51:25.782963 20336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:25.783973 20341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4315ebb0b43a41309f23145e433fbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } } }
I20260929 17:51:25.784088 20341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:25.784848 20342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4315ebb0b43a41309f23145e433fbaac. Latest consensus state: current_term: 3 leader_uuid: "4315ebb0b43a41309f23145e433fbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4315ebb0b43a41309f23145e433fbaac" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44201 } } }
I20260929 17:51:25.785219 20342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:25.787698 20356 catalog_manager.cc:1259] Loaded cluster ID: b7c29504a15f47329e7551ebf6c58197
I20260929 17:51:25.787753 20356 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: loading cluster ID for follower catalog manager: success
I20260929 17:51:25.790325 20356 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4315ebb0b43a41309f23145e433fbaac: acquiring CA information for follower catalog manager: success
F20260929 17:51:25.791520 20356 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5fa5c989ad google::LogMessage::Fail() at ??:0
@ 0x7f5fa5c9b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5fa5c984de google::LogMessage::Flush() at ??:0
@ 0x7f5fa5c99a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5fa895c18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5fb05fd989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f5fb05da023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f5fb05da54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f5fb05d3652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f5fb05d29e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f5fb0619541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5fa7edb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5fa951f6db start_thread at ??:0
@ 0x7f5fa479661f clone at ??:0
W20260929 17:51:25.874042 20338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:25.874334 20338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:25.874383 20338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:25.878691 20338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:25.878770 20338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:25.878832 20338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:25.884333 20338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:36501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=39753
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:25.885438 20338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:25.886468 20338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:25.896091 20376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:25.896122 20377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:25.896158 20379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:25.896584 20338 server_base.cc:1034] running on GCE node
I20260929 17:51:25.896971 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:25.897634 20338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:25.898821 20338 hybrid_clock.cc:648] HybridClock initialized: now 1790704285898797 us; error 41 us; skew 500 ppm
Sep 29 17:51:25 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704285, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:25.901628 20338 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:51:25.902560 20338 webserver.cc:460] Webserver started at http://127.3.183.193:39753/ using document root <none> and password file <none>
I20260929 17:51:25.903090 20338 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:25.903155 20338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:25.907346 20338 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:51:25.909650 20386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:25.911052 20338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:51:25.911172 20338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "a405a510e8ff49e7a5e7089746685b13"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "bb7b62f64061b7341d74f2053448e78a"
server_key_iv: "95c4d8eb9b49b33f7cd085b488364adb"
server_key_version: "encryptionkey@0"
I20260929 17:51:25.911724 20338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:25.928668 20338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:25.931951 20338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:25.932147 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:25.932770 20338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:25.933885 20393 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:51:25.939553 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:51:25.939646 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 17:51:25.939733 20338 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:51:25.941610 20338 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:51:25.941684 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:51:25.941982 20393 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Bootstrap starting.
I20260929 17:51:25.953459 20338 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:36501
I20260929 17:51:25.953651 20500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:36501 every 8 connection(s)
I20260929 17:51:25.954874 20338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:25.962090 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 20338
I20260929 17:51:25.962381 20393 log.cc:824] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:25.962598 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:45043
--local_ip_for_outbound_sockets=127.3.183.194
--tserver_master_addrs=127.3.183.254:44201
--webserver_port=34983
--webserver_interface=127.3.183.194
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:51:25.971449 20393 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:51:25.971947 20393 tablet_bootstrap.cc:492] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Bootstrap complete.
I20260929 17:51:25.973306 20393 ts_tablet_manager.cc:1397] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260929 17:51:25.976666 20393 raft_consensus.cc:348] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:25.977267 20393 raft_consensus.cc:729] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a405a510e8ff49e7a5e7089746685b13, State: Initialized, Role: FOLLOWER
I20260929 17:51:25.977803 20393 consensus_queue.cc:260] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:25.978894 20393 ts_tablet_manager.cc:1428] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 17:51:25.990304 20503 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:51:25.957284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:35617)
0929 17:51:25.957812 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:25.957867 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:25.958710 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:25.989547 (+ 30837us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":156,"threads_started":1}
W20260929 17:51:25.990763 20501 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:44201 (0 consecutive failures): Network error: Failed to ping master at 127.3.183.254:44201: Client connection negotiation failed: client connection to 127.3.183.254:44201: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:51:25.991437 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:25.991191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:54279)
0929 17:51:25.991310 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:25.991376 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 17:51:25.992022 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:25.991832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:45571)
0929 17:51:25.991926 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:25.991972 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:51:25.992165 20501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:51:26.079492 20505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:26.079753 20505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:26.079843 20505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:26.083804 20505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:26.083865 20505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:26.083943 20505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:26.089035 20505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:45043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=34983
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:26.090095 20505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:26.091131 20505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:26.100972 20514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:26.101066 20511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:26.101051 20505 server_base.cc:1034] running on GCE node
W20260929 17:51:26.100972 20512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:26.101655 20505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:26.102300 20505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:26.103466 20505 hybrid_clock.cc:648] HybridClock initialized: now 1790704286103439 us; error 40 us; skew 500 ppm
Sep 29 17:51:26 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704286, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:26.106245 20505 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:51:26.107126 20505 webserver.cc:460] Webserver started at http://127.3.183.194:34983/ using document root <none> and password file <none>
I20260929 17:51:26.107663 20505 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:26.107728 20505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:26.111555 20505 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:26.113585 20521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:26.114864 20505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:51:26.114965 20505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "8499e8b033894eedbc118a4dd0072142"
format_stamp: "Formatted at 2026-09-29 17:50:43 on dist-test-slave-wbpt"
server_key: "621621c4b56bf0a8841194c6d5ffc871"
server_key_iv: "85f8cda0d357a07810a47792fca24a40"
server_key_version: "encryptionkey@0"
I20260929 17:51:26.115509 20505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:26.129622 20505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:26.132743 20505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:26.132920 20505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:26.133512 20505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:26.134334 20505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:26.134389 20505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:26.134443 20505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:26.134469 20505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:26.142776 20505 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:45043
I20260929 17:51:26.142824 20634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:45043 every 8 connection(s)
I20260929 17:51:26.143769 20505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:26.147351 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:26.145890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:48373)
0929 17:51:26.146558 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:26.146665 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":114,"threads_started":1}
W20260929 17:51:26.147866 20635 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:44201 (0 consecutive failures): Network error: Failed to ping master at 127.3.183.254:44201: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
I20260929 17:51:26.148497 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:26.148231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:52423)
0929 17:51:26.148393 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:26.148443 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 17:51:26.149133 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:26.148926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:46711)
0929 17:51:26.149039 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:26.149083 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 17:51:26.149286 20635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 17:51:26.149750 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 20505
I20260929 17:51:26.150179 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:34301
--local_ip_for_outbound_sockets=127.3.183.195
--tserver_master_addrs=127.3.183.254:44201
--webserver_port=42811
--webserver_interface=127.3.183.195
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:26.259549 20638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:26.259863 20638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:26.259908 20638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:26.263929 20638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:26.264008 20638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:26.264068 20638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:26.269505 20638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:34301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=42811
--tserver_master_addrs=127.3.183.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:26.270583 20638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:26.271549 20638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:26.281044 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:26.281062 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:26.281034 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:26.281356 20638 server_base.cc:1034] running on GCE node
I20260929 17:51:26.281832 20638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:26.282464 20638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:26.283653 20638 hybrid_clock.cc:648] HybridClock initialized: now 1790704286283635 us; error 30 us; skew 500 ppm
Sep 29 17:51:26 dist-test-slave-wbpt krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704286, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:26.286347 20638 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:51:26.287184 20638 webserver.cc:460] Webserver started at http://127.3.183.195:42811/ using document root <none> and password file <none>
I20260929 17:51:26.287717 20638 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:26.287810 20638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:26.291668 20638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 17:51:26.293754 20653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:26.294993 20638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:26.295133 20638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "ec11cab700dc4e83a2d4a9c8d499aea0"
format_stamp: "Formatted at 2026-09-29 17:50:44 on dist-test-slave-wbpt"
server_key: "c1105037b7fc304a3a92c25f22fbddeb"
server_key_iv: "c08967eb649b5148c31d740df8c97e93"
server_key_version: "encryptionkey@0"
I20260929 17:51:26.295665 20638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:26.331892 20638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:26.335081 20638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:26.335278 20638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:26.335932 20638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:26.336773 20638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:26.336820 20638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:26.336874 20638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:26.336901 20638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:26.345467 20638 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:34301
I20260929 17:51:26.345515 20766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:34301 every 8 connection(s)
I20260929 17:51:26.346544 20638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:51:26.350123 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:26.348628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:33165)
0929 17:51:26.349296 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:26.349407 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":114,"threads_started":1}
W20260929 17:51:26.350689 20767 heartbeater.cc:644] Failed to heartbeat to 127.3.183.254:44201 (0 consecutive failures): Network error: Failed to ping master at 127.3.183.254:44201: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
I20260929 17:51:26.351284 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:26.351072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:34399)
0929 17:51:26.351186 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:26.351231 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 17:51:26.351907 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:26.351697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:35627)
0929 17:51:26.351811 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:26.351859 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:51:26.352043 20767 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 17:51:26.356829 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 20638
I20260929 17:51:26.993227 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:26.992753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:34701)
0929 17:51:26.993034 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:26.993100 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
I20260929 17:51:27.150209 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:27.149813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:60421)
0929 17:51:27.150041 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:27.150099 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260929 17:51:27.352922 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:27.352508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:58415)
0929 17:51:27.352761 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:27.352816 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":68,"threads_started":1}
I20260929 17:51:27.503629 20774 raft_consensus.cc:482] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:27.503882 20774 raft_consensus.cc:504] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:27.505108 20774 leader_election.cc:288] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301), 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
Sep 29 17:51:27 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704285, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.195@KRBTEST.COM
Sep 29 17:51:27 dist-test-slave-wbpt krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704285, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.194@KRBTEST.COM
I20260929 17:51:27.519325 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:27.505970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:55811 (local address 127.3.183.194:45043)
0929 17:51:27.506290 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:27.506310 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:27.506342 (+ 32us) server_negotiation.cc:407] Connection header received
0929 17:51:27.506997 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:27.507002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:27.507282 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:27.507565 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:27.508754 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.509724 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.510988 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.511456 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.511585 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:27.514643 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:27.514649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:27.514663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:27.514699 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:27.516939 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:27.517475 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:27.517480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:27.517486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:27.517616 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:27.517910 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:27.517912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:27.517914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:27.518313 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:27.518432 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:27.518843 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 17:51:27.519098 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
I20260929 17:51:27.519317 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:27.505511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:48899 (local address 127.3.183.195:34301)
0929 17:51:27.505978 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:27.506008 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:27.506057 (+ 49us) server_negotiation.cc:407] Connection header received
0929 17:51:27.506811 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:27.506819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:27.507165 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:27.507431 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:27.508761 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.509755 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.511038 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.511492 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.511651 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:27.514300 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:27.514307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:27.514324 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:27.514357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:27.516540 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:27.517134 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:27.517141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:27.517146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:27.517291 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:27.517713 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:27.517716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:27.517717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:27.518174 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:27.518414 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:27.518845 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 17:51:27.519093 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":118,"threads_started":1}
I20260929 17:51:27.519356 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:27.505742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45043 (local address 127.3.183.193:55811)
0929 17:51:27.505999 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:27.506011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:27.506093 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:27.507553 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:27.507559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:27.508121 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:27.508591 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:27.508607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.509877 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.509881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:27.510867 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:27.510878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.511576 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.511580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:27.511805 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:27.512392 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:27.512408 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:27.514459 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:27.517083 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:27.517087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:27.517089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:27.517326 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:27.517713 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:27.517715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:27.517717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:27.517822 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:27.518406 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:27.518409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:27.518558 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 17:51:27.518723 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 17:51:27.519131 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":271,"spinlock_wait_cycles":89344,"thread_start_us":63,"threads_started":1}
I20260929 17:51:27.519359 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:27.505309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34301 (local address 127.3.183.193:48899)
0929 17:51:27.505588 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:27.505602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:27.505702 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:27.507551 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:27.507557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:27.508120 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:27.508591 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:27.508607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.509913 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.509916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:27.510889 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:27.510898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:27.511616 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:27.511619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:27.511801 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:27.512432 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:27.512451 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:27.514044 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:27.516661 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:27.516667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:27.516680 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:27.516999 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:27.517418 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:27.517421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:27.517423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:27.517616 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:27.518309 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:27.518316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:27.518558 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 17:51:27.518723 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 17:51:27.519060 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":154,"spinlock_wait_cycles":5120,"thread_start_us":118,"threads_started":1}
W20260929 17:51:27.521873 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
W20260929 17:51:27.522112 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
I20260929 17:51:27.522172 20389 leader_election.cc:302] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a405a510e8ff49e7a5e7089746685b13; no voters: 8499e8b033894eedbc118a4dd0072142, ec11cab700dc4e83a2d4a9c8d499aea0
I20260929 17:51:27.522432 20774 raft_consensus.cc:2726] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:51:27.994355 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:27.994030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:42461)
0929 17:51:27.994204 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:27.994272 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 17:51:28.151341 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:28.150923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:49305)
0929 17:51:28.151173 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:28.151227 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260929 17:51:28.354033 20780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:28.353612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:33407)
0929 17:51:28.353868 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:28.353944 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260929 17:51:28.995566 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:28.995101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:50801)
0929 17:51:28.995367 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:28.995425 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
I20260929 17:51:29.152599 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:29.152131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:53515)
0929 17:51:29.152409 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:29.152477 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260929 17:51:29.245316 20783 raft_consensus.cc:482] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:29.245473 20783 raft_consensus.cc:504] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:29.245850 20783 leader_election.cc:288] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301), 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
W20260929 17:51:29.246585 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
W20260929 17:51:29.246733 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
I20260929 17:51:29.246778 20389 leader_election.cc:302] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a405a510e8ff49e7a5e7089746685b13; no voters: 8499e8b033894eedbc118a4dd0072142, ec11cab700dc4e83a2d4a9c8d499aea0
I20260929 17:51:29.246935 20783 raft_consensus.cc:2726] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:51:29.355155 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:29.354705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:51495)
0929 17:51:29.354973 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:29.355031 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
I20260929 17:51:29.996760 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:29.996316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:40777)
0929 17:51:29.996586 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:29.996657 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260929 17:51:30.153867 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:30.153383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:33333)
0929 17:51:30.153707 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:30.153770 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260929 17:51:30.356359 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:30.355953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:55659)
0929 17:51:30.356200 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:30.356264 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260929 17:51:30.998210 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:30.997742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:53737)
0929 17:51:30.998049 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:30.998112 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20260929 17:51:31.155231 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:31.154766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:36915)
0929 17:51:31.155066 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:31.155133 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260929 17:51:31.357635 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:31.357198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:45085)
0929 17:51:31.357472 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:31.357535 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260929 17:51:31.999550 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:31.999104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:51441)
0929 17:51:31.999382 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:31.999449 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260929 17:51:32.156567 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:32.156147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:38413)
0929 17:51:32.156413 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:32.156470 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260929 17:51:32.358809 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:32.358380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:51379)
0929 17:51:32.358641 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:32.358704 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260929 17:51:32.562569 20794 raft_consensus.cc:482] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:32.562697 20794 raft_consensus.cc:504] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:32.563125 20794 leader_election.cc:288] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301), 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
W20260929 17:51:32.563992 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
W20260929 17:51:32.564109 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
I20260929 17:51:32.564157 20389 leader_election.cc:302] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a405a510e8ff49e7a5e7089746685b13; no voters: 8499e8b033894eedbc118a4dd0072142, ec11cab700dc4e83a2d4a9c8d499aea0
I20260929 17:51:32.564311 20794 raft_consensus.cc:2726] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:51:33.000801 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:33.000377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:34093)
0929 17:51:33.000649 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:33.000702 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260929 17:51:33.157799 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:33.157322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:40695)
0929 17:51:33.157642 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:33.157702 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":78,"threads_started":1}
I20260929 17:51:33.360126 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:33.359617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:39317)
0929 17:51:33.359967 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:33.360033 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":78,"threads_started":1}
I20260929 17:51:34.001956 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:34.001515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:44189)
0929 17:51:34.001776 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:34.001834 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":69,"threads_started":1}
I20260929 17:51:34.158994 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:34.158553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:57705)
0929 17:51:34.158818 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:34.158874 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260929 17:51:34.361303 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:34.360875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:53355)
0929 17:51:34.361137 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:34.361199 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260929 17:51:35.003100 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:35.002644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:37615)
0929 17:51:35.002925 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:35.002983 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260929 17:51:35.124212 20802 raft_consensus.cc:482] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:35.124346 20802 raft_consensus.cc:504] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec11cab700dc4e83a2d4a9c8d499aea0" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34301 } } peers { permanent_uuid: "a405a510e8ff49e7a5e7089746685b13" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 36501 } } peers { permanent_uuid: "8499e8b033894eedbc118a4dd0072142" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45043 } }
I20260929 17:51:35.124778 20802 leader_election.cc:288] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301), 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043)
W20260929 17:51:35.125586 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8499e8b033894eedbc118a4dd0072142 (127.3.183.194:45043): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
W20260929 17:51:35.125697 20389 leader_election.cc:341] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ec11cab700dc4e83a2d4a9c8d499aea0 (127.3.183.195:34301): Not found: Tablet not found: 5e45e1db59fd4f9cb0b0f24d9082ccc9
I20260929 17:51:35.125741 20389 leader_election.cc:302] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a405a510e8ff49e7a5e7089746685b13; no voters: 8499e8b033894eedbc118a4dd0072142, ec11cab700dc4e83a2d4a9c8d499aea0
I20260929 17:51:35.125921 20802 raft_consensus.cc:2726] T 5e45e1db59fd4f9cb0b0f24d9082ccc9 P a405a510e8ff49e7a5e7089746685b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:51:35.160344 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:35.159910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:51415)
0929 17:51:35.160174 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:35.160238 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260929 17:51:35.362707 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:35.362228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:48085)
0929 17:51:35.362544 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:35.362610 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260929 17:51:36.004578 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:36.004056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:42633)
0929 17:51:36.004347 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:36.004418 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260929 17:51:36.161648 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:36.161164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:58685)
0929 17:51:36.161489 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:36.161553 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
I20260929 17:51:36.363914 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:36.363457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:47877)
0929 17:51:36.363724 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:36.363816 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260929 17:51:37.005831 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:37.005378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:55127)
0929 17:51:37.005665 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:37.005729 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260929 17:51:37.162940 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:37.162484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:53447)
0929 17:51:37.162755 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:37.162819 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260929 17:51:37.365031 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:37.364566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:46751)
0929 17:51:37.364865 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:37.364928 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260929 17:51:38.007091 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:38.006658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:45977)
0929 17:51:38.006933 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:38.006996 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260929 17:51:38.164390 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:38.163834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:38823)
0929 17:51:38.164171 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:38.164250 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
I20260929 17:51:38.366204 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:38.365761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:43475)
0929 17:51:38.366044 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:38.366106 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
I20260929 17:51:39.008566 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:39.007986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:48533)
0929 17:51:39.008312 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:39.008435 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260929 17:51:39.165511 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:39.165096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:40753)
0929 17:51:39.165356 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:39.165414 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260929 17:51:39.367368 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:39.366940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:52881)
0929 17:51:39.367213 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:39.367275 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260929 17:51:40.009775 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:40.009355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:41783)
0929 17:51:40.009624 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:40.009678 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":103,"threads_started":1}
I20260929 17:51:40.166769 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:40.166293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:47019)
0929 17:51:40.166601 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:40.166665 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
I20260929 17:51:40.368721 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:40.368198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:46477)
0929 17:51:40.368542 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:40.368616 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260929 17:51:41.011264 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:41.010803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.193:50687)
0929 17:51:41.011105 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:41.011167 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260929 17:51:41.168129 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:41.167669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.194:38861)
0929 17:51:41.167967 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:41.168030 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":72,"threads_started":1}
I20260929 17:51:41.364194 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 20338
I20260929 17:51:41.369865 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 20505
I20260929 17:51:41.370572 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:41.369736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:44201 (local address 127.3.183.195:44057)
0929 17:51:41.370394 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:41.370462 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.254:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260929 17:51:41.374917 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 20638
2026-09-29T17:51:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59142 ms)
[----------] 30 tests from SecurityITest (242599 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:41 dist-test-slave-wbpt krb5kdc[20825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:41 dist-test-slave-wbpt krb5kdc[20825](info): set up 2 sockets
Sep 29 17:51:41 dist-test-slave-wbpt krb5kdc[20825](info): commencing operation
krb5kdc: starting...
W20260929 17:51:42.331645 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:42 dist-test-slave-wbpt krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:51:42.452968 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38971
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:42.565572 20838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:42.565898 20838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:42.565965 20838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:42.570138 20838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:42.570222 20838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:42.570247 20838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:42.570268 20838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:42.570287 20838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:42.575883 20838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:38971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:42.577088 20838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:42.578214 20838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:42.586833 20843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:42.586879 20846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:42.586833 20844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:42.587162 20838 server_base.cc:1034] running on GCE node
I20260929 17:51:42.587755 20838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:42.588992 20838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:42.590149 20838 hybrid_clock.cc:648] HybridClock initialized: now 1790704302590135 us; error 31 us; skew 500 ppm
Sep 29 17:51:42 dist-test-slave-wbpt krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704302, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:42.592944 20838 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:51:42.593827 20838 webserver.cc:460] Webserver started at http://127.3.183.254:44343/ using document root <none> and password file <none>
I20260929 17:51:42.594381 20838 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:42.594447 20838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:42.594686 20838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:42.596776 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "ccd7269368ba49b4a033f66f9fba68a5"
format_stamp: "Formatted at 2026-09-29 17:51:42 on dist-test-slave-wbpt"
server_key: "abcdbe494b8974c18e2fb1b4b6285d6a"
server_key_iv: "e167f677879306e935bb9305ce7ab0a1"
server_key_version: "encryptionkey@0"
I20260929 17:51:42.597319 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "ccd7269368ba49b4a033f66f9fba68a5"
format_stamp: "Formatted at 2026-09-29 17:51:42 on dist-test-slave-wbpt"
server_key: "abcdbe494b8974c18e2fb1b4b6285d6a"
server_key_iv: "e167f677879306e935bb9305ce7ab0a1"
server_key_version: "encryptionkey@0"
I20260929 17:51:42.601155 20838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:51:42.603513 20853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:42.604768 20838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:42.604885 20838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "ccd7269368ba49b4a033f66f9fba68a5"
format_stamp: "Formatted at 2026-09-29 17:51:42 on dist-test-slave-wbpt"
server_key: "abcdbe494b8974c18e2fb1b4b6285d6a"
server_key_iv: "e167f677879306e935bb9305ce7ab0a1"
server_key_version: "encryptionkey@0"
I20260929 17:51:42.604984 20838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:42.633600 20838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:42.636898 20838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:42.637085 20838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:42.643800 20838 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:38971
I20260929 17:51:42.643872 20905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:38971 every 8 connection(s)
I20260929 17:51:42.644771 20838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:42.647636 20906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:42.649730 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 20838
I20260929 17:51:42.649879 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:42.650158 3807 external_mini_cluster.cc:1428] Setting key 81e7946361a35eeba4059b9e9c027740
I20260929 17:51:42.654438 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5: Bootstrap starting.
I20260929 17:51:42.656746 20906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:42.657512 20906 log.cc:824] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:51:42 dist-test-slave-wbpt krb5kdc[20825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:42.659462 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5: No bootstrap required, opened a new log
I20260929 17:51:42.663090 20906 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd7269368ba49b4a033f66f9fba68a5" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38971 } }
I20260929 17:51:42.663365 20906 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:42.663447 20906 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd7269368ba49b4a033f66f9fba68a5, State: Initialized, Role: FOLLOWER
I20260929 17:51:42.663933 20906 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd7269368ba49b4a033f66f9fba68a5" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38971 } }
I20260929 17:51:42.664059 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:42.651618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.3.183.254:38971)
0929 17:51:42.652091 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:42.652106 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:42.652142 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:51:42.652805 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:42.652823 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:42.653164 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:42.653473 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:42.654260 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:42.655494 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:42.656469 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:42.656990 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:42.657161 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:42.659573 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:42.659579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:42.659597 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:42.659628 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:42.661220 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:42.661649 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:42.661655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:42.661660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:42.661772 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:42.662041 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:42.662045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:42.662046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:42.662435 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:42.662631 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:42.662893 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 17:51:42.663188 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":163,"threads_started":1}
I20260929 17:51:42.664068 20906 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:42.664160 20906 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:42.664253 20906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:42.665375 20906 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd7269368ba49b4a033f66f9fba68a5" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38971 } }
I20260929 17:51:42.665709 20906 leader_election.cc:302] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd7269368ba49b4a033f66f9fba68a5; no voters:
I20260929 17:51:42.666018 20906 leader_election.cc:288] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:42.666107 20911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:42.666344 20911 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [term 1 LEADER]: Becoming Leader. State: Replica: ccd7269368ba49b4a033f66f9fba68a5, State: Running, Role: LEADER
I20260929 17:51:42.666635 20911 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd7269368ba49b4a033f66f9fba68a5" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38971 } }
I20260929 17:51:42.667374 20906 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:42.667819 20911 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccd7269368ba49b4a033f66f9fba68a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd7269368ba49b4a033f66f9fba68a5" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38971 } } }
I20260929 17:51:42.668008 20911 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:42.667980 20912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd7269368ba49b4a033f66f9fba68a5. Latest consensus state: current_term: 1 leader_uuid: "ccd7269368ba49b4a033f66f9fba68a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd7269368ba49b4a033f66f9fba68a5" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38971 } } }
I20260929 17:51:42.668217 20912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:42.668432 20916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:42.671432 20916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:42.677541 20916 catalog_manager.cc:1347] Generated new cluster ID: f8e1af2c27264daf80d5816af5f29336
I20260929 17:51:42.677695 20916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:42.684176 20916 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:42.685463 20916 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:42.696732 20916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5: Generated new TSK 0
I20260929 17:51:42.697588 20916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:51:42.734402 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38971
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:42.872817 20932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:42.873090 20932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:42.873133 20932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:42.877234 20932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:42.877300 20932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:42.877360 20932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:42.882858 20932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:42.884001 20932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:42.885087 20932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:42.894816 20938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:42.894860 20932 server_base.cc:1034] running on GCE node
W20260929 17:51:42.894797 20937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:42.894797 20940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:42.895402 20932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:42.896086 20932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:42.897277 20932 hybrid_clock.cc:648] HybridClock initialized: now 1790704302897238 us; error 49 us; skew 500 ppm
Sep 29 17:51:42 dist-test-slave-wbpt krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704302, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:42.899962 20932 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:51:42.900779 20932 webserver.cc:460] Webserver started at http://127.3.183.193:43387/ using document root <none> and password file <none>
I20260929 17:51:42.901317 20932 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:42.901379 20932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:42.901567 20932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:42.903641 20932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5"
format_stamp: "Formatted at 2026-09-29 17:51:42 on dist-test-slave-wbpt"
server_key: "a704225b0408f4e2d21fb3a81d21ebd2"
server_key_iv: "24dbe974c7e6a1f8637afc42fa7505c0"
server_key_version: "encryptionkey@0"
I20260929 17:51:42.904203 20932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5"
format_stamp: "Formatted at 2026-09-29 17:51:42 on dist-test-slave-wbpt"
server_key: "a704225b0408f4e2d21fb3a81d21ebd2"
server_key_iv: "24dbe974c7e6a1f8637afc42fa7505c0"
server_key_version: "encryptionkey@0"
I20260929 17:51:42.908149 20932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:51:42.910753 20947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:42.912212 20932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:42.912318 20932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5"
format_stamp: "Formatted at 2026-09-29 17:51:42 on dist-test-slave-wbpt"
server_key: "a704225b0408f4e2d21fb3a81d21ebd2"
server_key_iv: "24dbe974c7e6a1f8637afc42fa7505c0"
server_key_version: "encryptionkey@0"
I20260929 17:51:42.912415 20932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:42.934293 20932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:42.937902 20932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:42.938072 20932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:42.938654 20932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:42.939574 20932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:42.939628 20932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:42.939675 20932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:42.939692 20932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:42.948457 20932 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:44735
I20260929 17:51:42.948490 21060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:44735 every 8 connection(s)
I20260929 17:51:42.949339 20932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:42.951958 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 20932
I20260929 17:51:42.952111 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:51:42.952431 3807 external_mini_cluster.cc:1428] Setting key 8d2e08712e22dec8f8359982370bc1f8
Sep 29 17:51:42 dist-test-slave-wbpt krb5kdc[20825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704302, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:51:42.964128 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:42.951119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:57999 (local address 127.3.183.254:38971)
0929 17:51:42.951288 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:42.951293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:42.952221 (+ 928us) server_negotiation.cc:407] Connection header received
0929 17:51:42.953753 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:42.953757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:42.953802 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:42.953881 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:42.955462 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:42.956073 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:42.957473 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:42.957814 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:42.957920 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:42.960342 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:42.960347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:42.960350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:42.960388 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:42.961858 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:42.962480 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:42.962483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:42.962484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:42.962575 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:42.962967 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:42.962970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:42.962971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:42.963179 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:42.963273 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:42.963839 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 17:51:42.963974 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:51:42.965059 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:42.951519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38971 (local address 127.3.183.193:57999)
0929 17:51:42.952048 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:42.952112 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:42.953535 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:42.954081 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:42.954090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:42.954635 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:42.955307 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:42.955326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:42.956198 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:42.956201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:42.957340 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:42.957350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:42.957945 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:42.957947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:42.958144 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:42.958706 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:42.958723 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:42.960170 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:42.962047 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:42.962052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:42.962066 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:42.962366 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:42.962704 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:42.962707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:42.962708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:42.962859 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:42.963282 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:42.963288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:42.963575 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 17:51:42.963738 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 17:51:42.964082 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":113,"threads_started":1}
I20260929 17:51:42.966367 21061 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38971
I20260929 17:51:42.966607 21061 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:42.967197 21061 heartbeater.cc:511] Master 127.3.183.254:38971 requested a full tablet report, sending...
I20260929 17:51:42.968961 20869 ts_manager.cc:194] Registered new tserver with Master: f0b8b9240a6f496fb1b27bbf8c7fa0b5 (127.3.183.193:44735)
I20260929 17:51:42.970633 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:57999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:51:42.984282 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38971
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:43.098978 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:43.099242 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:43.099301 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:43.103592 21066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:43.103659 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:43.103736 21066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:43.109103 21066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:43.110181 21066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:43.111202 21066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:43.121251 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:43.121326 21072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:43.121294 21066 server_base.cc:1034] running on GCE node
W20260929 17:51:43.121251 21071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:43.122020 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:43.122686 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:43.123860 21066 hybrid_clock.cc:648] HybridClock initialized: now 1790704303123840 us; error 34 us; skew 500 ppm
Sep 29 17:51:43 dist-test-slave-wbpt krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:43.126821 21066 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:51:43.127807 21066 webserver.cc:460] Webserver started at http://127.3.183.194:34485/ using document root <none> and password file <none>
I20260929 17:51:43.128394 21066 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:43.128464 21066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:43.128659 21066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:43.130812 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "757e6134606e40a2b287d8e91404ba86"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "5c46c49509e8eb6a72b4ca5b9ff05069"
server_key_iv: "c3bb70626c771bf14f69e6726303e8db"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.131404 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "757e6134606e40a2b287d8e91404ba86"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "5c46c49509e8eb6a72b4ca5b9ff05069"
server_key_iv: "c3bb70626c771bf14f69e6726303e8db"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.135247 21066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:51:43.137785 21081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:43.139070 21066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:43.139182 21066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "757e6134606e40a2b287d8e91404ba86"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "5c46c49509e8eb6a72b4ca5b9ff05069"
server_key_iv: "c3bb70626c771bf14f69e6726303e8db"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.139288 21066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:43.162587 21066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:43.165828 21066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:43.165992 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:43.166560 21066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:43.167343 21066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:43.167389 21066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:43.167429 21066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:43.167449 21066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:43.176031 21066 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:33899
I20260929 17:51:43.176074 21194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:33899 every 8 connection(s)
I20260929 17:51:43.176931 21066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:43.181221 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 21066
I20260929 17:51:43.181331 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:51:43.181612 3807 external_mini_cluster.cc:1428] Setting key 766ceebf23c2c140589ee071b5da7a43
Sep 29 17:51:43 dist-test-slave-wbpt krb5kdc[20825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:51:43.191247 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.178783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:47931 (local address 127.3.183.254:38971)
0929 17:51:43.178948 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.178952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.179857 (+ 905us) server_negotiation.cc:407] Connection header received
0929 17:51:43.181337 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.181341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.181440 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.181581 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:43.182968 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.183701 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.184926 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.185263 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.185368 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.187941 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:43.187944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:43.187946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:43.187970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:43.189313 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:43.189787 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:43.189790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:43.189791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:43.189864 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:43.190184 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:43.190187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:43.190188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:43.190362 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:43.190442 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.190958 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.191082 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 17:51:43.192153 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.179182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38971 (local address 127.3.183.194:47931)
0929 17:51:43.179661 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.179707 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:43.181048 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:43.181782 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:43.181793 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:43.182321 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:43.182825 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.182840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.183875 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.183878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.184799 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.184810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.185417 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.185420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.185572 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.186135 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:43.186154 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:43.187754 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:43.189421 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:43.189427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:43.189443 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:43.189688 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:43.189941 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:43.189944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:43.189946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:43.190090 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:43.190460 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:43.190466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:43.190723 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 17:51:43.190882 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 17:51:43.191215 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
I20260929 17:51:43.193339 21195 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38971
I20260929 17:51:43.193586 21195 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:43.194155 21195 heartbeater.cc:511] Master 127.3.183.254:38971 requested a full tablet report, sending...
I20260929 17:51:43.195277 20869 ts_manager.cc:194] Registered new tserver with Master: 757e6134606e40a2b287d8e91404ba86 (127.3.183.194:33899)
I20260929 17:51:43.195935 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:47931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:51:43.210495 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38971
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:43.317411 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:43.317663 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:43.317706 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:43.321724 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:43.321786 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:43.321853 21200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:43.327060 21200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:43.328138 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:43.329139 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:43.338575 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:43.338613 21200 server_base.cc:1034] running on GCE node
W20260929 17:51:43.338584 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:43.338596 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:43.339286 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:43.339975 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:43.341150 21200 hybrid_clock.cc:648] HybridClock initialized: now 1790704303341129 us; error 29 us; skew 500 ppm
Sep 29 17:51:43 dist-test-slave-wbpt krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:43.344149 21200 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:51:43.345149 21200 webserver.cc:460] Webserver started at http://127.3.183.195:40903/ using document root <none> and password file <none>
I20260929 17:51:43.345741 21200 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:43.345816 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:43.346037 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:43.348245 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "00965ac85cdf4159a6efcc106ed1ee46"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "0c7f783995e8edc4dad0f2c308d9f8a7"
server_key_iv: "fbb5c519ba5903a8eb557583fe987a98"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.348834 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "00965ac85cdf4159a6efcc106ed1ee46"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "0c7f783995e8edc4dad0f2c308d9f8a7"
server_key_iv: "fbb5c519ba5903a8eb557583fe987a98"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.352624 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:51:43.355175 21215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:43.356551 21200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:43.356662 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "00965ac85cdf4159a6efcc106ed1ee46"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "0c7f783995e8edc4dad0f2c308d9f8a7"
server_key_iv: "fbb5c519ba5903a8eb557583fe987a98"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.356766 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:43.366154 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:43.369275 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:43.369450 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:43.370029 21200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:43.370846 21200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:43.370894 21200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:43.370945 21200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:43.370975 21200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:43.379284 21200 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:44581
I20260929 17:51:43.379324 21328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:44581 every 8 connection(s)
I20260929 17:51:43.380194 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:51:43.386585 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 21200
I20260929 17:51:43.386718 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:51:43.387055 3807 external_mini_cluster.cc:1428] Setting key 26555213bfc2c7eef0fad8e922f3d28d
Sep 29 17:51:43 dist-test-slave-wbpt krb5kdc[20825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:43.394053 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.381933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:34129 (local address 127.3.183.254:38971)
0929 17:51:43.382105 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.382110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.382987 (+ 877us) server_negotiation.cc:407] Connection header received
0929 17:51:43.384359 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.384363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.384417 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.384512 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:43.385649 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.386196 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.387716 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.388051 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.388144 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.390705 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:43.390708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:43.390710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:43.390738 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:43.392252 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:43.392683 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:43.392685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:43.392687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:43.392771 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:43.393048 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:43.393051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:43.393052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:43.393237 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:43.393323 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.393799 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.393931 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:51:43.394850 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.382389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38971 (local address 127.3.183.195:34129)
0929 17:51:43.382833 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.382875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:43.384123 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:43.384653 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:43.384660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:43.385010 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:43.385494 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.385508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.386313 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.386317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.387591 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.387601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.388206 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.388210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.388388 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.389063 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:43.389084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:43.390548 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:43.392348 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:43.392352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:43.392364 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:43.392604 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:43.392862 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:43.392864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:43.392865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:43.392984 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:43.393311 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:43.393316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:43.393563 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 17:51:43.393718 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 17:51:43.393975 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260929 17:51:43.395983 21329 heartbeater.cc:347] Connected to a master server at 127.3.183.254:38971
I20260929 17:51:43.396200 21329 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:43.396667 21329 heartbeater.cc:511] Master 127.3.183.254:38971 requested a full tablet report, sending...
I20260929 17:51:43.397670 20869 ts_manager.cc:194] Registered new tserver with Master: 00965ac85cdf4159a6efcc106ed1ee46 (127.3.183.195:44581)
I20260929 17:51:43.398236 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:34129
I20260929 17:51:43.400784 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:51:43.409461 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.402626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.3.183.254:38971)
0929 17:51:43.402773 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.402777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.402962 (+ 185us) server_negotiation.cc:407] Connection header received
0929 17:51:43.403004 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.403006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.403045 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.403110 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:43.403771 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.404312 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.405120 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.405333 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.405414 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.406735 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:43.406738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:43.406740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:43.406762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:43.408125 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:43.408464 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:43.408467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:43.408468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:43.408530 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:43.408715 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:43.408717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:43.408718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:43.408943 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:43.409047 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.409206 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.409292 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 17:51:43.412201 20869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:51:43.415136 20869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:51:43.429452 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.424450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.3.183.195:44581)
0929 17:51:43.424910 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.424921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.424940 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:51:43.425009 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.425014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.425223 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.425705 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:43.425837 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.426637 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.428105 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.428816 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.428911 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.429070 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.429211 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.429305 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":247,"threads_started":1}
I20260929 17:51:43.429456 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.423765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:44581 (local address 127.0.0.1:42538)
0929 17:51:43.424237 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.424287 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:43.424424 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:43.425391 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:43.425397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:43.425421 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:43.425579 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.425587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.426752 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.426755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.427976 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.427986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.428907 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.428910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.429027 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.429068 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 17:51:43.429178 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 17:51:43.429299 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":3,"spinlock_wait_cycles":5120,"thread_start_us":68,"threads_started":1}
I20260929 17:51:43.429831 21341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.423960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:33899 (local address 127.0.0.1:43112)
0929 17:51:43.424237 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.424285 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:43.424424 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:43.425827 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:43.425830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:43.425845 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:43.425931 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.425936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.427258 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.427261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.428411 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.428420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.429467 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.429471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.429572 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.429595 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:51:43.429642 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 17:51:43.429706 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":22,"spinlock_wait_cycles":8320,"thread_start_us":107,"threads_started":1}
I20260929 17:51:43.430121 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.424463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:44735 (local address 127.0.0.1:38354)
0929 17:51:43.424685 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.424697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:43.424774 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:43.425946 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:43.425949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:43.425966 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:43.426061 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.426068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.427533 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.427536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.428749 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.428759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.429779 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.429782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.429870 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.429894 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:51:43.429943 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 17:51:43.430007 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260929 17:51:43.430500 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.425081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.3.183.194:33899)
0929 17:51:43.425394 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.425404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.425427 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:51:43.425499 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.425504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.425692 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.425838 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:43.426029 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.427124 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.428540 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.429342 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.430087 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.430224 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.430270 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.430351 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
I20260929 17:51:43.430732 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.425198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.3.183.193:44735)
0929 17:51:43.425557 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.425571 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.425590 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:51:43.425650 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.425655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.425807 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.426205 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:43.426347 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.427379 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.428874 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.429656 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.430358 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.430471 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.430511 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.430594 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":84,"threads_started":1}
I20260929 17:51:43.433183 20995 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8c51cb6a6f48069e349132e8b5b41e (DEFAULT_TABLE table=test-table [id=edf299e491874b25a91d49be6d744ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:43.433284 21263 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8c51cb6a6f48069e349132e8b5b41e (DEFAULT_TABLE table=test-table [id=edf299e491874b25a91d49be6d744ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:43.433284 21129 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8c51cb6a6f48069e349132e8b5b41e (DEFAULT_TABLE table=test-table [id=edf299e491874b25a91d49be6d744ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:43.434187 20995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8c51cb6a6f48069e349132e8b5b41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:43.434303 21263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8c51cb6a6f48069e349132e8b5b41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:43.434298 21129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8c51cb6a6f48069e349132e8b5b41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:43.440117 21346 tablet_bootstrap.cc:492] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5: Bootstrap starting.
I20260929 17:51:43.441911 21348 tablet_bootstrap.cc:492] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Bootstrap starting.
I20260929 17:51:43.442480 21346 tablet_bootstrap.cc:654] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:43.443519 21346 log.cc:824] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:43.444065 21348 tablet_bootstrap.cc:654] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:43.444139 21347 tablet_bootstrap.cc:492] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: Bootstrap starting.
I20260929 17:51:43.444868 21348 log.cc:824] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:43.445336 21346 tablet_bootstrap.cc:492] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5: No bootstrap required, opened a new log
I20260929 17:51:43.445534 21346 ts_tablet_manager.cc:1397] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 17:51:43.446062 21347 tablet_bootstrap.cc:654] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:43.446961 21347 log.cc:824] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:43.447252 21348 tablet_bootstrap.cc:492] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: No bootstrap required, opened a new log
I20260929 17:51:43.447436 21348 ts_tablet_manager.cc:1397] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:51:43.448472 21347 tablet_bootstrap.cc:492] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: No bootstrap required, opened a new log
I20260929 17:51:43.448629 21347 ts_tablet_manager.cc:1397] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:51:43.449383 21346 raft_consensus.cc:348] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.449644 21346 raft_consensus.cc:374] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:43.449698 21346 raft_consensus.cc:729] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b8b9240a6f496fb1b27bbf8c7fa0b5, State: Initialized, Role: FOLLOWER
I20260929 17:51:43.450155 21346 consensus_queue.cc:260] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.451138 21346 ts_tablet_manager.cc:1428] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 17:51:43.451092 21348 raft_consensus.cc:348] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.451350 21348 raft_consensus.cc:374] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:43.451406 21348 raft_consensus.cc:729] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00965ac85cdf4159a6efcc106ed1ee46, State: Initialized, Role: FOLLOWER
I20260929 17:51:43.451488 21061 heartbeater.cc:503] Master 127.3.183.254:38971 was elected leader, sending a full tablet report...
I20260929 17:51:43.451859 21348 consensus_queue.cc:260] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.451745 21347 raft_consensus.cc:348] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.452003 21347 raft_consensus.cc:374] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:43.452054 21347 raft_consensus.cc:729] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757e6134606e40a2b287d8e91404ba86, State: Initialized, Role: FOLLOWER
I20260929 17:51:43.452440 21347 consensus_queue.cc:260] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.452787 21348 ts_tablet_manager.cc:1428] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 17:51:43.452818 21329 heartbeater.cc:503] Master 127.3.183.254:38971 was elected leader, sending a full tablet report...
I20260929 17:51:43.453230 21195 heartbeater.cc:503] Master 127.3.183.254:38971 was elected leader, sending a full tablet report...
I20260929 17:51:43.453306 21347 ts_tablet_manager.cc:1428] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:51:43.583125 21354 raft_consensus.cc:482] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:43.583331 21354 raft_consensus.cc:504] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.584710 21354 leader_election.cc:288] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00965ac85cdf4159a6efcc106ed1ee46 (127.3.183.195:44581), f0b8b9240a6f496fb1b27bbf8c7fa0b5 (127.3.183.193:44735)
I20260929 17:51:43.588078 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.584896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:44581 (local address 127.3.183.194:49729)
0929 17:51:43.585015 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.585030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:43.585115 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:43.585371 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:43.585373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:43.585392 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:43.585508 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.585514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.586237 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.586241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.587028 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.587037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.587665 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.587669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.587773 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.587813 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 17:51:43.587878 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:51:43.587947 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 17:51:43.588078 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.585014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:49729 (local address 127.3.183.195:44581)
0929 17:51:43.585133 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.585138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.585157 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:51:43.585213 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.585216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.585260 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.585336 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:43.585623 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.586131 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.587156 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.587559 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.587665 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.587717 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.587908 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.587975 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:51:43.588621 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.585168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:44735 (local address 127.3.183.194:58327)
0929 17:51:43.585590 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.585600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:43.585685 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:43.585903 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:43.585905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:43.585920 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:43.586028 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.586034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.586690 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.586693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.587446 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:43.587463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.588259 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.588262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:43.588332 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.588351 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:51:43.588424 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 17:51:43.588497 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":131,"threads_started":1}
I20260929 17:51:43.588639 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.585290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:58327 (local address 127.3.183.193:44735)
0929 17:51:43.585413 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.585418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.585683 (+ 265us) server_negotiation.cc:407] Connection header received
0929 17:51:43.585781 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.585784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.585830 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.585890 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:43.586127 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.586597 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.587570 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.588026 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.588127 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.588173 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.588449 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.588525 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:51:43.588840 21283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8c51cb6a6f48069e349132e8b5b41e" candidate_uuid: "757e6134606e40a2b287d8e91404ba86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00965ac85cdf4159a6efcc106ed1ee46" is_pre_election: true
I20260929 17:51:43.589167 21283 raft_consensus.cc:2454] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757e6134606e40a2b287d8e91404ba86 in term 0.
I20260929 17:51:43.589254 21015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8c51cb6a6f48069e349132e8b5b41e" candidate_uuid: "757e6134606e40a2b287d8e91404ba86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" is_pre_election: true
I20260929 17:51:43.589507 21015 raft_consensus.cc:2454] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757e6134606e40a2b287d8e91404ba86 in term 0.
I20260929 17:51:43.589627 21083 leader_election.cc:302] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00965ac85cdf4159a6efcc106ed1ee46, 757e6134606e40a2b287d8e91404ba86; no voters:
I20260929 17:51:43.589882 21354 raft_consensus.cc:2781] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:51:43.589967 21354 raft_consensus.cc:482] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:51:43.590015 21354 raft_consensus.cc:3037] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:43.590996 21354 raft_consensus.cc:504] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.591285 21354 leader_election.cc:288] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [CANDIDATE]: Term 1 election: Requested vote from peers 00965ac85cdf4159a6efcc106ed1ee46 (127.3.183.195:44581), f0b8b9240a6f496fb1b27bbf8c7fa0b5 (127.3.183.193:44735)
I20260929 17:51:43.591619 21283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8c51cb6a6f48069e349132e8b5b41e" candidate_uuid: "757e6134606e40a2b287d8e91404ba86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00965ac85cdf4159a6efcc106ed1ee46"
I20260929 17:51:43.591639 21015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8c51cb6a6f48069e349132e8b5b41e" candidate_uuid: "757e6134606e40a2b287d8e91404ba86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5"
I20260929 17:51:43.591740 21283 raft_consensus.cc:3037] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:43.591740 21015 raft_consensus.cc:3037] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:43.592989 21015 raft_consensus.cc:2454] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757e6134606e40a2b287d8e91404ba86 in term 1.
I20260929 17:51:43.592989 21283 raft_consensus.cc:2454] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757e6134606e40a2b287d8e91404ba86 in term 1.
I20260929 17:51:43.593273 21083 leader_election.cc:302] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00965ac85cdf4159a6efcc106ed1ee46, 757e6134606e40a2b287d8e91404ba86; no voters:
I20260929 17:51:43.593422 21354 raft_consensus.cc:2781] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:43.593647 21354 raft_consensus.cc:686] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 1 LEADER]: Becoming Leader. State: Replica: 757e6134606e40a2b287d8e91404ba86, State: Running, Role: LEADER
I20260929 17:51:43.593966 21354 consensus_queue.cc:237] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } }
I20260929 17:51:43.597100 20870 catalog_manager.cc:5495] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 757e6134606e40a2b287d8e91404ba86 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "757e6134606e40a2b287d8e91404ba86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757e6134606e40a2b287d8e91404ba86" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 } health_report { overall_health: UNKNOWN } } }
I20260929 17:51:43.629165 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.625937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.3.183.194:33899)
0929 17:51:43.626057 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.626060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.626073 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:51:43.626147 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.626149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.626202 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.626344 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:43.626485 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.626983 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.627804 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.628025 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.628109 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.628407 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:43.628838 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:51:43.628939 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.628996 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.629049 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 17:51:43.631078 21330 tablet.cc:2369] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:51:43.635145 21283 raft_consensus.cc:1261] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 1 FOLLOWER]: Refusing update from remote peer 757e6134606e40a2b287d8e91404ba86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:43.635145 21015 raft_consensus.cc:1261] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 1 FOLLOWER]: Refusing update from remote peer 757e6134606e40a2b287d8e91404ba86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:43.635896 21354 consensus_queue.cc:1035] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00965ac85cdf4159a6efcc106ed1ee46" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:43.636101 21356 consensus_queue.cc:1035] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0b8b9240a6f496fb1b27bbf8c7fa0b5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:43.643566 21361 mvcc.cc:205] Tried to move back new op lower bound from 7334724827682254848 to 7334724827523362816. Current Snapshot: MvccSnapshot[applied={T|T < 7334724827682254848}]
I20260929 17:51:43.652640 20870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:43.652822 20870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:43.655560 20870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccd7269368ba49b4a033f66f9fba68a5: Sending DeleteTablet for 3 replicas of tablet 6c8c51cb6a6f48069e349132e8b5b41e
I20260929 17:51:43.656635 21263 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8c51cb6a6f48069e349132e8b5b41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:43 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:42538
I20260929 17:51:43.656635 20995 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8c51cb6a6f48069e349132e8b5b41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:43 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:38354
I20260929 17:51:43.657086 20994 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8c51cb6a6f48069e349132e8b5b41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:43 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:38354
I20260929 17:51:43.657775 21369 tablet_replica.cc:330] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5: stopping tablet replica
I20260929 17:51:43.657742 21129 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8c51cb6a6f48069e349132e8b5b41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:43 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:43112
I20260929 17:51:43.657891 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 20932
I20260929 17:51:43.658078 21369 raft_consensus.cc:2229] T 6c8c51cb6a6f48069e349132e8b5b41e P f0b8b9240a6f496fb1b27bbf8c7fa0b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:43.658543 21370 tablet_replica.cc:330] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: stopping tablet replica
I20260929 17:51:43.659085 21370 raft_consensus.cc:2229] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:43.659368 21370 raft_consensus.cc:2258] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:43.659736 21371 tablet_replica.cc:330] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: stopping tablet replica
I20260929 17:51:43.660041 21371 raft_consensus.cc:2229] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:51:43.660351 21371 raft_consensus.cc:2258] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:43.661617 21370 ts_tablet_manager.cc:1905] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:43.664142 21370 ts_tablet_manager.cc:1918] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:43.664242 21370 log.cc:1196] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/6c8c51cb6a6f48069e349132e8b5b41e
I20260929 17:51:43.664564 21370 ts_tablet_manager.cc:1939] T 6c8c51cb6a6f48069e349132e8b5b41e P 00965ac85cdf4159a6efcc106ed1ee46: Deleting consensus metadata
I20260929 17:51:43.665012 21371 ts_tablet_manager.cc:1905] T 6c8c51cb6a6f48069e349132e8b5b41e P 757e6134606e40a2b287d8e91404ba86: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:51:43.665771 20855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:44735 (error 108)
I20260929 17:51:43.665977 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 21066
I20260929 17:51:43.667011 20855 catalog_manager.cc:4841] TS 00965ac85cdf4159a6efcc106ed1ee46 (127.3.183.195:44581): tablet 6c8c51cb6a6f48069e349132e8b5b41e (table test-table [id=edf299e491874b25a91d49be6d744ee4]) successfully deleted
I20260929 17:51:43.667870 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.667575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:44735 (local address 127.0.0.1:38370)
0929 17:51:43.667691 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.667765 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:44735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 17:51:43.668102 20855 catalog_manager.cc:4584] TS f0b8b9240a6f496fb1b27bbf8c7fa0b5 (127.3.183.193:44735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c8c51cb6a6f48069e349132e8b5b41e: Network error: Client connection negotiation failed: client connection to 127.3.183.193:44735: connect: Connection refused (error 111)
I20260929 17:51:43.671504 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 21200
I20260929 17:51:43.672240 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.671726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:33899 (local address 127.0.0.1:43122)
0929 17:51:43.672095 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:43.672148 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
I20260929 17:51:43.676744 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 20838
2026-09-29T17:51:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T17:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:43Z Disabled control of system clock
I20260929 17:51:43.706765 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:33305
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:33305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:51:43.816010 21376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:43.816303 21376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:43.816339 21376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:43.820369 21376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:43.820429 21376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:43.820443 21376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:43.820458 21376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:43.820472 21376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:43.825488 21376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:33305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:33305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:43.826690 21376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:43.828163 21376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:43.836870 21381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:43.836930 21376 server_base.cc:1034] running on GCE node
W20260929 17:51:43.836864 21384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:43.836864 21382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:43.837672 21376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:43.838912 21376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:43.840112 21376 hybrid_clock.cc:648] HybridClock initialized: now 1790704303840094 us; error 33 us; skew 500 ppm
I20260929 17:51:43.842248 21376 webserver.cc:460] Webserver started at http://127.3.183.254:32999/ using document root <none> and password file <none>
I20260929 17:51:43.842849 21376 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:43.842928 21376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:43.843137 21376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:43.845364 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "a05c589276254c7082aa45b8660e925c"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "fef647c95b3f893c35af4290575ec7ee"
server_key_iv: "89e0473f08d757532484e974222ff667"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.845956 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "a05c589276254c7082aa45b8660e925c"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "fef647c95b3f893c35af4290575ec7ee"
server_key_iv: "89e0473f08d757532484e974222ff667"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.849736 21376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 17:51:43.852133 21390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:43.853439 21376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:43.853572 21376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "a05c589276254c7082aa45b8660e925c"
format_stamp: "Formatted at 2026-09-29 17:51:43 on dist-test-slave-wbpt"
server_key: "fef647c95b3f893c35af4290575ec7ee"
server_key_iv: "89e0473f08d757532484e974222ff667"
server_key_version: "encryptionkey@0"
I20260929 17:51:43.853695 21376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:43.872859 21376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:43.873600 21376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:43.873806 21376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:43.881080 21376 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:33305
I20260929 17:51:43.881094 21442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:33305 every 8 connection(s)
I20260929 17:51:43.882097 21376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:43.882580 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 21376
I20260929 17:51:43.882720 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:43.883033 3807 external_mini_cluster.cc:1428] Setting key d4dc6de37115a3161f8568ba7d74edc4
I20260929 17:51:43.885599 21443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:43.891992 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c: Bootstrap starting.
I20260929 17:51:43.892575 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:43.884453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.3.183.254:33305)
0929 17:51:43.885067 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:43.885084 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:43.885122 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:51:43.885807 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:43.885825 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:43.886213 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:43.886501 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:43.886844 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.888064 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.889265 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:43.889747 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:43.889883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:43.890742 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:43.890747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:43.890760 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:43.891171 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:43.891228 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:43.891249 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:43.891324 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:43.891479 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 17:51:43.891738 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20260929 17:51:43.894346 21443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:43.895146 21443 log.cc:824] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:43.897028 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c: No bootstrap required, opened a new log
I20260929 17:51:43.900431 21443 raft_consensus.cc:348] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c589276254c7082aa45b8660e925c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33305 } }
I20260929 17:51:43.900657 21443 raft_consensus.cc:374] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:43.900693 21443 raft_consensus.cc:729] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05c589276254c7082aa45b8660e925c, State: Initialized, Role: FOLLOWER
I20260929 17:51:43.901129 21443 consensus_queue.cc:260] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c589276254c7082aa45b8660e925c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33305 } }
I20260929 17:51:43.901253 21443 raft_consensus.cc:388] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:43.901314 21443 raft_consensus.cc:482] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:43.901401 21443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:43.902276 21443 raft_consensus.cc:504] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c589276254c7082aa45b8660e925c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33305 } }
I20260929 17:51:43.902590 21443 leader_election.cc:302] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a05c589276254c7082aa45b8660e925c; no voters:
I20260929 17:51:43.902868 21443 leader_election.cc:288] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:43.903017 21448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:43.903213 21448 raft_consensus.cc:686] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [term 1 LEADER]: Becoming Leader. State: Replica: a05c589276254c7082aa45b8660e925c, State: Running, Role: LEADER
I20260929 17:51:43.903564 21448 consensus_queue.cc:237] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c589276254c7082aa45b8660e925c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33305 } }
I20260929 17:51:43.904068 21443 sys_catalog.cc:564] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:43.905198 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a05c589276254c7082aa45b8660e925c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c589276254c7082aa45b8660e925c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33305 } } }
I20260929 17:51:43.905324 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:43.905622 21450 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a05c589276254c7082aa45b8660e925c. Latest consensus state: current_term: 1 leader_uuid: "a05c589276254c7082aa45b8660e925c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c589276254c7082aa45b8660e925c" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 33305 } } }
I20260929 17:51:43.905710 21450 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:43.905849 21457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:43.908600 21457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:43.914363 21457 catalog_manager.cc:1347] Generated new cluster ID: 60240c6025214209b2daa98a76afcbe3
I20260929 17:51:43.914456 21457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:43.927814 21457 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:43.928846 21457 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:43.937618 21457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c: Generated new TSK 0
I20260929 17:51:43.938323 21457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:43.944797 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33305
--builtin_ntp_servers=127.3.183.212:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:51:44.072247 21467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:44.072536 21467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:44.072592 21467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:44.076651 21467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:44.076725 21467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:44.076797 21467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:44.082334 21467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:44.083396 21467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:44.084743 21467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:44.094400 21473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:44.094614 21475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:44.094403 21472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:44.094698 21467 server_base.cc:1034] running on GCE node
I20260929 17:51:44.095261 21467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:44.096017 21467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:44.097193 21467 hybrid_clock.cc:648] HybridClock initialized: now 1790704304097171 us; error 35 us; skew 500 ppm
I20260929 17:51:44.102991 21467 webserver.cc:460] Webserver started at http://127.3.183.193:34361/ using document root <none> and password file <none>
I20260929 17:51:44.103616 21467 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:44.103693 21467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:44.103966 21467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:44.106220 21467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "a84af21bd41e4542870fe75d92b761e5"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "8f482b3b96f735739e2a4406294d6977"
server_key_iv: "f03e2268a8a4a2570d848669d7168362"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.106762 21467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "a84af21bd41e4542870fe75d92b761e5"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "8f482b3b96f735739e2a4406294d6977"
server_key_iv: "f03e2268a8a4a2570d848669d7168362"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.110525 21467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:51:44.113093 21481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.114300 21467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:51:44.114408 21467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "a84af21bd41e4542870fe75d92b761e5"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "8f482b3b96f735739e2a4406294d6977"
server_key_iv: "f03e2268a8a4a2570d848669d7168362"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.114511 21467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:44.128050 21467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:44.128813 21467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:44.129047 21467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:44.129684 21467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:44.130673 21467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:44.130728 21467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.130781 21467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:44.130810 21467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.140342 21467 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:38553
I20260929 17:51:44.140466 21594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:38553 every 8 connection(s)
I20260929 17:51:44.141335 21467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:44.142016 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 21467
I20260929 17:51:44.142153 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:51:44.142514 3807 external_mini_cluster.cc:1428] Setting key a5620111bcdd1f59b4006e2c0367435d
I20260929 17:51:44.144605 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33305
--builtin_ntp_servers=127.3.183.212:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:51:44.152719 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.143444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:39821 (local address 127.3.183.254:33305)
0929 17:51:44.143669 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.143674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.144928 (+ 1254us) server_negotiation.cc:407] Connection header received
0929 17:51:44.146575 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.146578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.146634 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.146714 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:44.147983 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.148564 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.150346 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.150613 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.150704 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.152136 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:44.152139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:44.152141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:44.152156 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:44.152178 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:44.152186 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:44.152264 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.152493 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.152588 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 17:51:44.153923 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.143998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33305 (local address 127.3.183.193:39821)
0929 17:51:44.144776 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.144834 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.146298 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.146937 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.146946 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.147066 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:44.147799 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.147821 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.148708 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.148712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.150187 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.150200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.150788 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.150793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.151013 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.151749 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:44.151751 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:44.151856 (+ 105us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:44.151963 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:44.152276 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:44.152285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:44.152292 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.152429 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.152782 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
I20260929 17:51:44.155471 21595 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33305
I20260929 17:51:44.155740 21595 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:44.156335 21595 heartbeater.cc:511] Master 127.3.183.254:33305 requested a full tablet report, sending...
I20260929 17:51:44.157994 21407 ts_manager.cc:194] Registered new tserver with Master: a84af21bd41e4542870fe75d92b761e5 (127.3.183.193:38553)
I20260929 17:51:44.159209 21407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.193:39821
W20260929 17:51:44.260578 21598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:44.260871 21598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:44.260914 21598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:44.265079 21598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:44.265157 21598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:44.265223 21598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:44.270139 21598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:44.271235 21598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:44.272701 21598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:44.282729 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:44.282809 21606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:44.282729 21604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:44.282877 21598 server_base.cc:1034] running on GCE node
I20260929 17:51:44.283560 21598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:44.284267 21598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:44.285429 21598 hybrid_clock.cc:648] HybridClock initialized: now 1790704304285405 us; error 36 us; skew 500 ppm
I20260929 17:51:44.287222 21598 webserver.cc:460] Webserver started at http://127.3.183.194:39057/ using document root <none> and password file <none>
I20260929 17:51:44.287771 21598 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:44.287882 21598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:44.288100 21598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:44.290347 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "cd92c33563c442eeb7bd06a97855ee30"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "c0d79ec2dfdc1430aa614a6449b674fb"
server_key_iv: "cb15e609307958900d0842b18171a16e"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.290917 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "cd92c33563c442eeb7bd06a97855ee30"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "c0d79ec2dfdc1430aa614a6449b674fb"
server_key_iv: "cb15e609307958900d0842b18171a16e"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.294531 21598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 17:51:44.297011 21612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.298249 21598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:51:44.298360 21598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "cd92c33563c442eeb7bd06a97855ee30"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "c0d79ec2dfdc1430aa614a6449b674fb"
server_key_iv: "cb15e609307958900d0842b18171a16e"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.298460 21598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:44.322134 21598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:44.322870 21598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:44.323047 21598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:44.323635 21598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:44.324630 21598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:44.324678 21598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.324733 21598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:44.324760 21598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.332820 21598 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:42151
I20260929 17:51:44.332845 21725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:42151 every 8 connection(s)
I20260929 17:51:44.333748 21598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:44.341791 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 21598
I20260929 17:51:44.341907 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:51:44.342182 3807 external_mini_cluster.cc:1428] Setting key eafdb4e8f5f63e1a804b604e639c5ed1
I20260929 17:51:44.342720 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.335496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:51209 (local address 127.3.183.254:33305)
0929 17:51:44.335626 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.335629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.336785 (+ 1156us) server_negotiation.cc:407] Connection header received
0929 17:51:44.338156 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.338159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.338219 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.338290 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:44.339193 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.339711 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.340878 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.341091 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.341188 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.342248 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:44.342252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:44.342255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:44.342269 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:44.342293 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:44.342302 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:44.342363 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.342502 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.342574 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:51:44.343745 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.335940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33305 (local address 127.3.183.194:51209)
0929 17:51:44.336632 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.336678 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.337944 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.338474 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.338481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.338567 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:44.339054 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.339067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.339860 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.339863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.340754 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.340763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.341229 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.341233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.341404 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.341998 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:44.342000 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:44.342045 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:44.342108 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:44.342372 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:44.342381 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:44.342387 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.342496 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.342822 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260929 17:51:44.344400 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33305
--builtin_ntp_servers=127.3.183.212:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:51:44.344973 21726 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33305
I20260929 17:51:44.345208 21726 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:44.345734 21726 heartbeater.cc:511] Master 127.3.183.254:33305 requested a full tablet report, sending...
I20260929 17:51:44.346783 21407 ts_manager.cc:194] Registered new tserver with Master: cd92c33563c442eeb7bd06a97855ee30 (127.3.183.194:42151)
I20260929 17:51:44.347362 21407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.194:51209
W20260929 17:51:44.451702 21729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:44.452013 21729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:44.452052 21729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:44.456036 21729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:44.456094 21729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:44.456153 21729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:44.460968 21729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:44.461983 21729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:44.463317 21729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:44.472538 21735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:44.472559 21737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:44.472711 21729 server_base.cc:1034] running on GCE node
W20260929 17:51:44.472524 21734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:44.473285 21729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:44.473927 21729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:44.475097 21729 hybrid_clock.cc:648] HybridClock initialized: now 1790704304475078 us; error 35 us; skew 500 ppm
I20260929 17:51:44.476941 21729 webserver.cc:460] Webserver started at http://127.3.183.195:44963/ using document root <none> and password file <none>
I20260929 17:51:44.477516 21729 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:44.477591 21729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:44.477803 21729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:44.479902 21729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "b385511eeb8b40478f47b40de8ab9f0d"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "322ac4e5483c1f26c8f9e08bb12b4261"
server_key_iv: "394b013b4737d0921fffea594077baf8"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.480475 21729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "b385511eeb8b40478f47b40de8ab9f0d"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "322ac4e5483c1f26c8f9e08bb12b4261"
server_key_iv: "394b013b4737d0921fffea594077baf8"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.484311 21729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:51:44.486688 21743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.487892 21729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:44.488001 21729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "b385511eeb8b40478f47b40de8ab9f0d"
format_stamp: "Formatted at 2026-09-29 17:51:44 on dist-test-slave-wbpt"
server_key: "322ac4e5483c1f26c8f9e08bb12b4261"
server_key_iv: "394b013b4737d0921fffea594077baf8"
server_key_version: "encryptionkey@0"
I20260929 17:51:44.488094 21729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:44.498191 21729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:44.498812 21729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:44.498974 21729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:44.499516 21729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:44.500480 21729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:44.500530 21729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.500581 21729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:44.500605 21729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:44.508697 21729 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:34291
I20260929 17:51:44.508783 21856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:34291 every 8 connection(s)
I20260929 17:51:44.509559 21729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:51:44.510345 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 21729
I20260929 17:51:44.510461 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:51:44.510730 3807 external_mini_cluster.cc:1428] Setting key 1800eecf6216350ce2d3caa19b01684b
I20260929 17:51:44.519848 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.512386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:39293 (local address 127.3.183.254:33305)
0929 17:51:44.512515 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.512518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.513807 (+ 1289us) server_negotiation.cc:407] Connection header received
0929 17:51:44.515160 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.515163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.515220 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.515289 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:44.516250 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.516776 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.517919 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.518185 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.518269 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.519353 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:44.519356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:44.519359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:44.519372 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:44.519392 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:44.519400 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:44.519458 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.519601 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.519665 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:51:44.520728 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.512972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:33305 (local address 127.3.183.195:39293)
0929 17:51:44.513669 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.513708 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.514951 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.515487 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.515495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.515582 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:44.516120 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.516133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.516898 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.516902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.517802 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.517811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.518332 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.518336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.518499 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.519069 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:44.519070 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:44.519130 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:44.519214 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:44.519467 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:44.519475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:44.519482 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.519590 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.519922 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260929 17:51:44.521809 21857 heartbeater.cc:347] Connected to a master server at 127.3.183.254:33305
I20260929 17:51:44.522046 21857 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:44.522567 21857 heartbeater.cc:511] Master 127.3.183.254:33305 requested a full tablet report, sending...
I20260929 17:51:44.523571 21407 ts_manager.cc:194] Registered new tserver with Master: b385511eeb8b40478f47b40de8ab9f0d (127.3.183.195:34291)
I20260929 17:51:44.524246 21407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.195:39293
I20260929 17:51:44.524883 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:51:44.529858 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.526714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.3.183.254:33305)
0929 17:51:44.526831 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.526835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.526988 (+ 153us) server_negotiation.cc:407] Connection header received
0929 17:51:44.527083 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.527085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.527122 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.527181 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:44.527380 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.527907 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.528695 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.528895 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.528968 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.529576 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:44.529579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:44.529581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:44.529591 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:44.529610 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:44.529617 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:44.529666 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.529714 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.529758 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:51:44.532639 21406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:51:44.535529 21406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:51:44.549432 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.543557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42151 (local address 127.0.0.1:37214)
0929 17:51:44.544658 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.544711 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.544901 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.545573 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.545576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.545608 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:44.545767 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.545772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.546734 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.546738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.548006 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.548016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.548877 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.548882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.548985 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.549030 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.549143 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.549261 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":92,"spinlock_wait_cycles":11392,"thread_start_us":78,"threads_started":1}
I20260929 17:51:44.549448 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.543692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.3.183.194:42151)
0929 17:51:44.544608 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.544620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.544793 (+ 173us) server_negotiation.cc:407] Connection header received
0929 17:51:44.545116 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.545121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.545287 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.545428 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:44.545893 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.546629 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.548114 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.548762 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.548867 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.549077 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.549182 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.549284 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":131,"threads_started":1}
I20260929 17:51:44.549903 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.543900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.3.183.193:38553)
0929 17:51:44.544409 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.544421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.544938 (+ 517us) server_negotiation.cc:407] Connection header received
0929 17:51:44.545059 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.545064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.545293 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.545432 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:44.545870 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.547005 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.548494 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.549107 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.549193 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.549355 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.549650 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.549740 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":145,"threads_started":1}
I20260929 17:51:44.549921 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.544226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:34291 (local address 127.0.0.1:41870)
0929 17:51:44.544908 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.544920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.544991 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.545435 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.545441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.545461 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:44.545587 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.545593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.547004 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.547007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.548321 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.548332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.549533 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.549537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.549632 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.549655 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.549723 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.549792 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":44,"thread_start_us":90,"threads_started":1}
I20260929 17:51:44.550014 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.544484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.3.183.195:34291)
0929 17:51:44.544785 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.544796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.545076 (+ 280us) server_negotiation.cc:407] Connection header received
0929 17:51:44.545147 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.545151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.545291 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.545424 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:44.545691 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.546863 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.548463 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.549197 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.549291 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.549469 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.549768 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.549860 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":127,"threads_started":1}
I20260929 17:51:44.550268 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.543600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:38553 (local address 127.0.0.1:35004)
0929 17:51:44.544829 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.544842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.544946 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.545573 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.545576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.545604 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:44.545751 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.545759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.547114 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.547117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.548331 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.548338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.549414 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.549417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.549497 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.549519 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.550071 (+ 552us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.550144 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":125,"spinlock_wait_cycles":4736,"thread_start_us":85,"threads_started":1}
I20260929 17:51:44.552969 21791 tablet_service.cc:1467] Processing CreateTablet for tablet c480e89cd9ba49a195f9cd5ee2068c08 (DEFAULT_TABLE table=test-table [id=d7c3c03da025452eb3873e20f8371deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:44.552982 21660 tablet_service.cc:1467] Processing CreateTablet for tablet c480e89cd9ba49a195f9cd5ee2068c08 (DEFAULT_TABLE table=test-table [id=d7c3c03da025452eb3873e20f8371deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:44.552982 21529 tablet_service.cc:1467] Processing CreateTablet for tablet c480e89cd9ba49a195f9cd5ee2068c08 (DEFAULT_TABLE table=test-table [id=d7c3c03da025452eb3873e20f8371deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:44.553900 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c480e89cd9ba49a195f9cd5ee2068c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:44.553957 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c480e89cd9ba49a195f9cd5ee2068c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:44.553956 21660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c480e89cd9ba49a195f9cd5ee2068c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:44.559975 21874 tablet_bootstrap.cc:492] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Bootstrap starting.
I20260929 17:51:44.561403 21875 tablet_bootstrap.cc:492] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: Bootstrap starting.
I20260929 17:51:44.562168 21874 tablet_bootstrap.cc:654] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:44.562884 21874 log.cc:824] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:44.563124 21875 tablet_bootstrap.cc:654] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:44.563522 21876 tablet_bootstrap.cc:492] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: Bootstrap starting.
I20260929 17:51:44.563866 21875 log.cc:824] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:44.564839 21874 tablet_bootstrap.cc:492] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: No bootstrap required, opened a new log
I20260929 17:51:44.565034 21874 ts_tablet_manager.cc:1397] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:51:44.565798 21875 tablet_bootstrap.cc:492] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: No bootstrap required, opened a new log
I20260929 17:51:44.565865 21876 tablet_bootstrap.cc:654] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:44.565975 21875 ts_tablet_manager.cc:1397] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:51:44.566520 21876 log.cc:824] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:44.568058 21876 tablet_bootstrap.cc:492] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: No bootstrap required, opened a new log
I20260929 17:51:44.568216 21876 ts_tablet_manager.cc:1397] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:51:44.569264 21874 raft_consensus.cc:348] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.569540 21874 raft_consensus.cc:374] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:44.569603 21874 raft_consensus.cc:729] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b385511eeb8b40478f47b40de8ab9f0d, State: Initialized, Role: FOLLOWER
I20260929 17:51:44.570098 21874 consensus_queue.cc:260] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.570041 21875 raft_consensus.cc:348] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.570292 21875 raft_consensus.cc:374] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:44.570339 21875 raft_consensus.cc:729] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84af21bd41e4542870fe75d92b761e5, State: Initialized, Role: FOLLOWER
I20260929 17:51:44.570799 21875 consensus_queue.cc:260] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.571106 21874 ts_tablet_manager.cc:1428] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:51:44.571446 21857 heartbeater.cc:503] Master 127.3.183.254:33305 was elected leader, sending a full tablet report...
I20260929 17:51:44.571436 21876 raft_consensus.cc:348] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.571657 21876 raft_consensus.cc:374] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:44.571702 21876 raft_consensus.cc:729] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd92c33563c442eeb7bd06a97855ee30, State: Initialized, Role: FOLLOWER
I20260929 17:51:44.571836 21875 ts_tablet_manager.cc:1428] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:51:44.572141 21595 heartbeater.cc:503] Master 127.3.183.254:33305 was elected leader, sending a full tablet report...
I20260929 17:51:44.572135 21876 consensus_queue.cc:260] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.572937 21726 heartbeater.cc:503] Master 127.3.183.254:33305 was elected leader, sending a full tablet report...
I20260929 17:51:44.574064 21876 ts_tablet_manager.cc:1428] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 17:51:44.584666 21727 tablet.cc:2369] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:51:44.642527 21596 tablet.cc:2369] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:51:44.760468 21858 tablet.cc:2369] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:51:44.968456 21880 raft_consensus.cc:482] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:44.968683 21880 raft_consensus.cc:504] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.969942 21880 leader_election.cc:288] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd92c33563c442eeb7bd06a97855ee30 (127.3.183.194:42151), a84af21bd41e4542870fe75d92b761e5 (127.3.183.193:38553)
I20260929 17:51:44.973675 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.970437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:50529 (local address 127.3.183.194:42151)
0929 17:51:44.970566 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.970570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.970585 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:51:44.970642 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.970645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.970699 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.970780 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:44.971115 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.971679 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.972755 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.973146 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.973241 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.973294 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.973474 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.973543 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:51:44.973688 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.970347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:52139 (local address 127.3.183.193:38553)
0929 17:51:44.970467 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.970471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.970782 (+ 311us) server_negotiation.cc:407] Connection header received
0929 17:51:44.970844 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.970847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.970896 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.971008 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:44.971283 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.971812 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.972799 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.973176 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.973289 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.973336 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.973501 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.973567 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:51:44.973688 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.970202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42151 (local address 127.3.183.195:50529)
0929 17:51:44.970339 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.970358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.970464 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.970830 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.970833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.970854 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:44.970978 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.970986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.971814 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.971817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.972627 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.972636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.973261 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.973264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.973360 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.973383 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.973438 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.973544 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 17:51:44.973709 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.970247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:38553 (local address 127.3.183.195:52139)
0929 17:51:44.970654 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:44.970667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:44.970771 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:44.971024 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:44.971026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:44.971045 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:44.971170 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.971177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.971915 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.971918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.972677 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:44.972687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.973285 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.973289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:44.973384 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.973401 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:51:44.973470 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:51:44.973539 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":3,"thread_start_us":142,"threads_started":1}
I20260929 17:51:44.974426 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c480e89cd9ba49a195f9cd5ee2068c08" candidate_uuid: "b385511eeb8b40478f47b40de8ab9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd92c33563c442eeb7bd06a97855ee30" is_pre_election: true
I20260929 17:51:44.974462 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c480e89cd9ba49a195f9cd5ee2068c08" candidate_uuid: "b385511eeb8b40478f47b40de8ab9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84af21bd41e4542870fe75d92b761e5" is_pre_election: true
I20260929 17:51:44.974756 21549 raft_consensus.cc:2454] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b385511eeb8b40478f47b40de8ab9f0d in term 0.
I20260929 17:51:44.974756 21680 raft_consensus.cc:2454] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b385511eeb8b40478f47b40de8ab9f0d in term 0.
I20260929 17:51:44.975260 21744 leader_election.cc:302] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84af21bd41e4542870fe75d92b761e5, b385511eeb8b40478f47b40de8ab9f0d; no voters:
I20260929 17:51:44.975561 21880 raft_consensus.cc:2781] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:51:44.975642 21880 raft_consensus.cc:482] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:51:44.975688 21880 raft_consensus.cc:3037] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:44.976936 21880 raft_consensus.cc:504] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.977272 21880 leader_election.cc:288] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [CANDIDATE]: Term 1 election: Requested vote from peers cd92c33563c442eeb7bd06a97855ee30 (127.3.183.194:42151), a84af21bd41e4542870fe75d92b761e5 (127.3.183.193:38553)
I20260929 17:51:44.977607 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c480e89cd9ba49a195f9cd5ee2068c08" candidate_uuid: "b385511eeb8b40478f47b40de8ab9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84af21bd41e4542870fe75d92b761e5"
I20260929 17:51:44.977635 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c480e89cd9ba49a195f9cd5ee2068c08" candidate_uuid: "b385511eeb8b40478f47b40de8ab9f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd92c33563c442eeb7bd06a97855ee30"
I20260929 17:51:44.977723 21549 raft_consensus.cc:3037] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:44.977744 21680 raft_consensus.cc:3037] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:44.978690 21680 raft_consensus.cc:2454] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b385511eeb8b40478f47b40de8ab9f0d in term 1.
I20260929 17:51:44.978722 21549 raft_consensus.cc:2454] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b385511eeb8b40478f47b40de8ab9f0d in term 1.
I20260929 17:51:44.978999 21745 leader_election.cc:302] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b385511eeb8b40478f47b40de8ab9f0d, cd92c33563c442eeb7bd06a97855ee30; no voters:
I20260929 17:51:44.979146 21880 raft_consensus.cc:2781] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:44.979337 21880 raft_consensus.cc:686] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 1 LEADER]: Becoming Leader. State: Replica: b385511eeb8b40478f47b40de8ab9f0d, State: Running, Role: LEADER
I20260929 17:51:44.979655 21880 consensus_queue.cc:237] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } }
I20260929 17:51:44.982646 21406 catalog_manager.cc:5495] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d reported cstate change: term changed from 0 to 1, leader changed from <none> to b385511eeb8b40478f47b40de8ab9f0d (127.3.183.195). New cstate: current_term: 1 leader_uuid: "b385511eeb8b40478f47b40de8ab9f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b385511eeb8b40478f47b40de8ab9f0d" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 34291 } health_report { overall_health: HEALTHY } } }
I20260929 17:51:44.998947 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:44.995715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.3.183.195:34291)
0929 17:51:44.995843 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:44.995847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:44.995859 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:51:44.995993 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:44.995995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:44.996046 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:44.996126 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:44.996289 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.996769 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.997626 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:44.997867 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:44.997950 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:44.998223 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:44.998626 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:51:44.998715 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:44.998765 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 17:51:44.998826 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:51:45.004668 21680 raft_consensus.cc:1261] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 1 FOLLOWER]: Refusing update from remote peer b385511eeb8b40478f47b40de8ab9f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:45.004659 21549 raft_consensus.cc:1261] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 1 FOLLOWER]: Refusing update from remote peer b385511eeb8b40478f47b40de8ab9f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:45.005376 21884 consensus_queue.cc:1035] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd92c33563c442eeb7bd06a97855ee30" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 42151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:45.005589 21880 consensus_queue.cc:1035] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84af21bd41e4542870fe75d92b761e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:45.012626 21888 mvcc.cc:205] Tried to move back new op lower bound from 7334724833292177408 to 7334724833199058944. Current Snapshot: MvccSnapshot[applied={T|T < 7334724833292177408}]
I20260929 17:51:45.013495 21887 mvcc.cc:205] Tried to move back new op lower bound from 7334724833292177408 to 7334724833199058944. Current Snapshot: MvccSnapshot[applied={T|T < 7334724833292177408}]
I20260929 17:51:45.021750 21407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:45.021947 21407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:45.024427 21407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a05c589276254c7082aa45b8660e925c: Sending DeleteTablet for 3 replicas of tablet c480e89cd9ba49a195f9cd5ee2068c08
I20260929 17:51:45.025168 21660 tablet_service.cc:1514] Processing DeleteTablet for tablet c480e89cd9ba49a195f9cd5ee2068c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:45 UTC) from {username='slave'} at 127.0.0.1:37214
I20260929 17:51:45.025185 21529 tablet_service.cc:1514] Processing DeleteTablet for tablet c480e89cd9ba49a195f9cd5ee2068c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:45 UTC) from {username='slave'} at 127.0.0.1:35004
I20260929 17:51:45.025270 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet c480e89cd9ba49a195f9cd5ee2068c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:45 UTC) from {username='slave'} at 127.0.0.1:41870
I20260929 17:51:45.025725 21897 tablet_replica.cc:330] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5: stopping tablet replica
I20260929 17:51:45.026058 21897 raft_consensus.cc:2229] T c480e89cd9ba49a195f9cd5ee2068c08 P a84af21bd41e4542870fe75d92b761e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:45.026350 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 21467
I20260929 17:51:45.030402 21899 tablet_replica.cc:330] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: stopping tablet replica
I20260929 17:51:45.030727 21899 raft_consensus.cc:2229] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:51:45.031098 21899 raft_consensus.cc:2258] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:45.031519 21898 tablet_replica.cc:330] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30: stopping tablet replica
I20260929 17:51:45.031874 21898 raft_consensus.cc:2229] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:45.032212 21898 raft_consensus.cc:2258] T c480e89cd9ba49a195f9cd5ee2068c08 P cd92c33563c442eeb7bd06a97855ee30 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 17:51:45.032217 21391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:38553 (error 108)
I20260929 17:51:45.032959 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 21598
I20260929 17:51:45.033689 21899 ts_tablet_manager.cc:1905] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:45.036736 21899 ts_tablet_manager.cc:1918] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:45.036844 21899 log.cc:1196] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/c480e89cd9ba49a195f9cd5ee2068c08
I20260929 17:51:45.037212 21899 ts_tablet_manager.cc:1939] T c480e89cd9ba49a195f9cd5ee2068c08 P b385511eeb8b40478f47b40de8ab9f0d: Deleting consensus metadata
I20260929 17:51:45.038543 21393 catalog_manager.cc:4841] TS b385511eeb8b40478f47b40de8ab9f0d (127.3.183.195:34291): tablet c480e89cd9ba49a195f9cd5ee2068c08 (table test-table [id=d7c3c03da025452eb3873e20f8371deb]) successfully deleted
I20260929 17:51:45.039137 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:45.038827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:38553 (local address 127.0.0.1:58264)
0929 17:51:45.038968 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:45.039046 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:38553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 17:51:45.039403 21391 catalog_manager.cc:4584] TS a84af21bd41e4542870fe75d92b761e5 (127.3.183.193:38553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c480e89cd9ba49a195f9cd5ee2068c08: Network error: Client connection negotiation failed: client connection to 127.3.183.193:38553: connect: Connection refused (error 111)
I20260929 17:51:45.039866 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 21729
W20260929 17:51:45.039987 21869 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:51:45.039661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:42151 (local address 127.0.0.1:37770)
0929 17:51:45.039773 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:45.039907 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:42151: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 17:51:45.044905 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 21376
2026-09-29T17:51:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1366 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:45 dist-test-slave-wbpt krb5kdc[21903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:45 dist-test-slave-wbpt krb5kdc[21903](info): set up 2 sockets
Sep 29 17:51:45 dist-test-slave-wbpt krb5kdc[21903](info): commencing operation
krb5kdc: starting...
W20260929 17:51:45.987661 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:51:46.108997 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:45911
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:46.219816 21916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:46.220090 21916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:46.220129 21916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:46.224166 21916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:46.224254 21916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:46.224279 21916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:46.224300 21916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:46.224321 21916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:46.229643 21916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:45911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:46.230664 21916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:46.231624 21916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:46.240001 21924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:46.240001 21922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:46.240198 21916 server_base.cc:1034] running on GCE node
W20260929 17:51:46.240031 21921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:46.240808 21916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:46.241921 21916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:46.243101 21916 hybrid_clock.cc:648] HybridClock initialized: now 1790704306243095 us; error 43 us; skew 500 ppm
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:46.245991 21916 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:51:46.246958 21916 webserver.cc:460] Webserver started at http://127.3.183.254:40127/ using document root <none> and password file <none>
I20260929 17:51:46.247500 21916 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:46.247574 21916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:46.247804 21916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:46.249891 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "63a83f265d1b47d7b05fb375ac77d794"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "1351121631831d7d1200d68de1e2dd52"
server_key_iv: "bca57dc63b81a08bf5c972023119cb95"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.250455 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "63a83f265d1b47d7b05fb375ac77d794"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "1351121631831d7d1200d68de1e2dd52"
server_key_iv: "bca57dc63b81a08bf5c972023119cb95"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.254269 21916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:51:46.256809 21931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.258093 21916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:46.258205 21916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "63a83f265d1b47d7b05fb375ac77d794"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "1351121631831d7d1200d68de1e2dd52"
server_key_iv: "bca57dc63b81a08bf5c972023119cb95"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.258303 21916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:46.276764 21916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:46.279902 21916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:46.280079 21916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:46.286959 21916 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:45911
I20260929 17:51:46.287020 21983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:45911 every 8 connection(s)
I20260929 17:51:46.287936 21916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:46.290750 21984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:46.295598 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 21916
I20260929 17:51:46.295826 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:46.296136 3807 external_mini_cluster.cc:1428] Setting key 397b383c1ba93757382afca7cbc8f778
I20260929 17:51:46.296720 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794: Bootstrap starting.
I20260929 17:51:46.299113 21984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:46.300009 21984 log.cc:824] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:46.302130 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794: No bootstrap required, opened a new log
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:46.306427 21984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a83f265d1b47d7b05fb375ac77d794" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45911 } }
I20260929 17:51:46.306692 21984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:46.306753 21984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a83f265d1b47d7b05fb375ac77d794, State: Initialized, Role: FOLLOWER
I20260929 17:51:46.307179 21984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a83f265d1b47d7b05fb375ac77d794" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45911 } }
I20260929 17:51:46.307308 21984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:46.307371 21984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:46.307466 21984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:46.308566 21984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a83f265d1b47d7b05fb375ac77d794" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45911 } }
I20260929 17:51:46.308895 21984 leader_election.cc:302] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a83f265d1b47d7b05fb375ac77d794; no voters:
I20260929 17:51:46.309222 21984 leader_election.cc:288] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:46.309278 21989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:46.309523 21989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [term 1 LEADER]: Becoming Leader. State: Replica: 63a83f265d1b47d7b05fb375ac77d794, State: Running, Role: LEADER
I20260929 17:51:46.309823 21989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a83f265d1b47d7b05fb375ac77d794" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45911 } }
I20260929 17:51:46.310418 21984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:46.310804 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:46.297563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.3.183.254:45911)
0929 17:51:46.297963 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:46.297979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:46.298014 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:51:46.298644 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:46.298661 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:46.299026 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:46.299345 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:46.300241 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.301408 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.302557 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.303014 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.303130 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:46.305811 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:46.305816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:46.305829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:46.305854 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:46.307591 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.308085 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.308090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.308094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.308211 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.308540 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.308543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.308544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.308976 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:46.309134 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:46.309499 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 17:51:46.309806 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260929 17:51:46.311033 21989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63a83f265d1b47d7b05fb375ac77d794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a83f265d1b47d7b05fb375ac77d794" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45911 } } }
I20260929 17:51:46.311128 21989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [sys.catalog]: This master's current role is: LEADER
W20260929 17:51:46.314973 22003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:51:46.315042 22003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:51:46.322642 21989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63a83f265d1b47d7b05fb375ac77d794. Latest consensus state: current_term: 1 leader_uuid: "63a83f265d1b47d7b05fb375ac77d794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a83f265d1b47d7b05fb375ac77d794" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 45911 } } }
I20260929 17:51:46.322767 22005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:46.322808 21989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:46.323900 22005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:46.330312 22005 catalog_manager.cc:1347] Generated new cluster ID: 0b4c4183d4c947a4a83c60bdc591fd39
I20260929 17:51:46.330462 22005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:46.337379 22005 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:46.338516 22005 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:46.346817 22005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794: Generated new TSK 0
I20260929 17:51:46.347533 22005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:51:46.377156 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45911
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:46.485550 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:46.485801 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:46.485859 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:46.489880 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:46.489940 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:46.490025 22010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:46.495142 22010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:46.496203 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:46.497247 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:46.506397 22018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:46.506512 22010 server_base.cc:1034] running on GCE node
W20260929 17:51:46.506389 22016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:46.506393 22015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:46.507035 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:46.507627 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:46.508772 22010 hybrid_clock.cc:648] HybridClock initialized: now 1790704306508753 us; error 53 us; skew 500 ppm
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:46.511443 22010 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:51:46.512391 22010 webserver.cc:460] Webserver started at http://127.3.183.193:35229/ using document root <none> and password file <none>
I20260929 17:51:46.512919 22010 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:46.513005 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:46.513204 22010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:46.515164 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "ef5d7182e08444eca980afb1ed07e207"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "76df5058952bc532cc751bb9254d1458"
server_key_iv: "0741777ba8d8f652382edb669ce3d383"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.515691 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "ef5d7182e08444eca980afb1ed07e207"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "76df5058952bc532cc751bb9254d1458"
server_key_iv: "0741777ba8d8f652382edb669ce3d383"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.519228 22010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:46.521554 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.522671 22010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:51:46.522774 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "ef5d7182e08444eca980afb1ed07e207"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "76df5058952bc532cc751bb9254d1458"
server_key_iv: "0741777ba8d8f652382edb669ce3d383"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.522862 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:46.537942 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:46.540700 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:46.540875 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:46.541443 22010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:46.542244 22010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:46.542290 22010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.542342 22010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:46.542371 22010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.550709 22010 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:44501
I20260929 17:51:46.550732 22138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:44501 every 8 connection(s)
I20260929 17:51:46.551568 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:46.553332 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22010
I20260929 17:51:46.553453 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:51:46.553774 3807 external_mini_cluster.cc:1428] Setting key 5cf57a72bf01ef18e65f31930f673e72
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:51:46.564779 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:46.553378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:46289 (local address 127.3.183.254:45911)
0929 17:51:46.553521 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:46.553525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:46.554537 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 17:51:46.555594 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:46.555597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:46.555646 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:46.555729 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:46.556964 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.557500 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.558640 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.558900 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.558990 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:46.561390 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:46.561392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:46.561394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:46.561416 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:46.562704 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.563207 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.563210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.563212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.563286 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.563592 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.563595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.563597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.563850 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:46.563935 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:46.564473 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 17:51:46.564615 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:51:46.565527 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:46.553861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45911 (local address 127.3.183.193:46289)
0929 17:51:46.554368 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:46.554430 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:46.555400 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:46.555922 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:46.555930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:46.556288 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:46.556814 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:46.556829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.557621 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.557624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:46.558517 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:46.558528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.559039 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.559043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:46.559200 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:46.559755 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:46.559771 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:46.561237 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:46.562859 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:46.562864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:46.562879 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:46.563117 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:46.563372 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:46.563375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:46.563377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:46.563508 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:46.563942 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:46.563948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:46.564197 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 17:51:46.564365 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 17:51:46.564680 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260929 17:51:46.566829 22139 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45911
I20260929 17:51:46.567067 22139 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:46.567559 22139 heartbeater.cc:511] Master 127.3.183.254:45911 requested a full tablet report, sending...
I20260929 17:51:46.569069 21948 ts_manager.cc:194] Registered new tserver with Master: ef5d7182e08444eca980afb1ed07e207 (127.3.183.193:44501)
I20260929 17:51:46.570168 21948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:46289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:51:46.581156 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45911
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:46.689685 22144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:46.689960 22144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:46.690003 22144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:46.694121 22144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:46.694190 22144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:46.694252 22144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:46.699571 22144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:46.700651 22144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:46.701656 22144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:46.711439 22152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:46.711607 22144 server_base.cc:1034] running on GCE node
W20260929 17:51:46.711462 22149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:46.711488 22150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:46.712224 22144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:46.712850 22144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:46.714023 22144 hybrid_clock.cc:648] HybridClock initialized: now 1790704306714007 us; error 28 us; skew 500 ppm
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:46.716867 22144 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:51:46.717793 22144 webserver.cc:460] Webserver started at http://127.3.183.194:35037/ using document root <none> and password file <none>
I20260929 17:51:46.718331 22144 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:46.718391 22144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:46.718596 22144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:46.720700 22144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "a506703c7f26460b87e77815680cb71b"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "8ec4d251d52bca0fffe7e5393e152445"
server_key_iv: "3ca816881e7ab85ab065c36da2b4217c"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.721240 22144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "a506703c7f26460b87e77815680cb71b"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "8ec4d251d52bca0fffe7e5393e152445"
server_key_iv: "3ca816881e7ab85ab065c36da2b4217c"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.724911 22144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 17:51:46.727409 22159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.728735 22144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:51:46.728868 22144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "a506703c7f26460b87e77815680cb71b"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "8ec4d251d52bca0fffe7e5393e152445"
server_key_iv: "3ca816881e7ab85ab065c36da2b4217c"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.728977 22144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:46.746129 22144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:46.749118 22144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:46.749313 22144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:46.749917 22144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:46.750737 22144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:46.750782 22144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.750833 22144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:46.750861 22144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.759159 22144 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:41731
I20260929 17:51:46.759222 22272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:41731 every 8 connection(s)
I20260929 17:51:46.760095 22144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:46.767553 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22144
I20260929 17:51:46.767705 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:51:46.768028 3807 external_mini_cluster.cc:1428] Setting key a4eef87bff01e025d5cdcf13143f0e6f
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:46.773293 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:46.761901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:47659 (local address 127.3.183.254:45911)
0929 17:51:46.762019 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:46.762022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:46.762957 (+ 935us) server_negotiation.cc:407] Connection header received
0929 17:51:46.764033 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:46.764036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:46.764082 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:46.764161 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:46.765358 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.765877 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.766979 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.767192 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.767271 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:46.770018 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:46.770021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:46.770023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:46.770066 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:46.771387 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.771861 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.771865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.771867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.771943 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.772259 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.772262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.772263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.772450 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:46.772532 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:46.773018 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 17:51:46.773132 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:51:46.774158 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:46.762266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45911 (local address 127.3.183.194:47659)
0929 17:51:46.762784 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:46.762845 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:46.763816 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:46.764314 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:46.764321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:46.764745 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:46.765233 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:46.765247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.765972 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.765974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:46.766866 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:46.766875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.767329 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.767333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:46.767523 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:46.768228 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:46.768248 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:46.769851 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:46.771493 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:46.771499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:46.771516 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:46.771749 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:46.772024 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:46.772027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:46.772029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:46.772165 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:46.772546 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:46.772552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:46.772779 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 17:51:46.772946 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 17:51:46.773265 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260929 17:51:46.775470 22273 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45911
I20260929 17:51:46.775710 22273 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:46.776237 22273 heartbeater.cc:511] Master 127.3.183.254:45911 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:51:46.777341 21948 ts_manager.cc:194] Registered new tserver with Master: a506703c7f26460b87e77815680cb71b (127.3.183.194:41731)
I20260929 17:51:46.777905 21948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:47659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:51:46.794584 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45911
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:46.903875 22278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:46.904119 22278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:46.904160 22278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:46.908128 22278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:46.908186 22278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:46.908254 22278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:46.913592 22278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:46.914602 22278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:46.915584 22278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:46.924881 22286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:46.924921 22284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:46.924921 22283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:46.925055 22278 server_base.cc:1034] running on GCE node
I20260929 17:51:46.925546 22278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:46.926214 22278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:46.927366 22278 hybrid_clock.cc:648] HybridClock initialized: now 1790704306927349 us; error 29 us; skew 500 ppm
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:46.930013 22278 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:51:46.930838 22278 webserver.cc:460] Webserver started at http://127.3.183.195:38585/ using document root <none> and password file <none>
I20260929 17:51:46.931373 22278 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:46.931435 22278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:46.931639 22278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:46.933717 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "b32297b965bc43d9bf67b70e2d00c80a"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "1811edef90fb8cede329ee325026da39"
server_key_iv: "414e0af3e9097ac2044e0f8998168cf8"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.934239 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "b32297b965bc43d9bf67b70e2d00c80a"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "1811edef90fb8cede329ee325026da39"
server_key_iv: "414e0af3e9097ac2044e0f8998168cf8"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.937848 22278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 17:51:46.940152 22293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.941288 22278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:46.941390 22278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "b32297b965bc43d9bf67b70e2d00c80a"
format_stamp: "Formatted at 2026-09-29 17:51:46 on dist-test-slave-wbpt"
server_key: "1811edef90fb8cede329ee325026da39"
server_key_iv: "414e0af3e9097ac2044e0f8998168cf8"
server_key_version: "encryptionkey@0"
I20260929 17:51:46.941525 22278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:46.965106 22278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:46.967972 22278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:46.968155 22278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:46.968753 22278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:46.969588 22278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:46.969635 22278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.969686 22278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:46.969714 22278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:46.977895 22278 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:35327
I20260929 17:51:46.977998 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:35327 every 8 connection(s)
I20260929 17:51:46.978891 22278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:51:46.980733 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22278
I20260929 17:51:46.980859 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:51:46.981141 3807 external_mini_cluster.cc:1428] Setting key 323bc7c5bad1a6c7c903c4187a0cf013
Sep 29 17:51:46 dist-test-slave-wbpt krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:46.992405 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:46.980751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:34433 (local address 127.3.183.254:45911)
0929 17:51:46.980898 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:46.980903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:46.981846 (+ 943us) server_negotiation.cc:407] Connection header received
0929 17:51:46.982911 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:46.982914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:46.982964 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:46.983018 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:46.984224 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.984763 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.986028 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.986246 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.986327 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:46.988824 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:46.988827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:46.988829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:46.988855 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:46.990216 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.990717 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.990721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.990722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.990805 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:46.991224 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:46.991227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:46.991228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:46.991424 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:46.991517 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:46.992159 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 17:51:46.992274 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:51:46.993284 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:46.981155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:45911 (local address 127.3.183.195:34433)
0929 17:51:46.981678 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:46.981741 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:46.982743 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:46.983208 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:46.983215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:46.983574 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:46.984101 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:46.984115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.984910 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.984913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:46.985905 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:46.985916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:46.986367 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:46.986370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:46.986552 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:46.987080 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:46.987096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:46.988645 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:46.990329 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:46.990334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:46.990347 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:46.990614 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:46.990920 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:46.990923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:46.990924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:46.991083 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:46.991581 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:46.991587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:46.991886 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 17:51:46.992087 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 17:51:46.992360 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":124,"threads_started":1}
I20260929 17:51:46.994513 22407 heartbeater.cc:347] Connected to a master server at 127.3.183.254:45911
I20260929 17:51:46.994763 22407 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:46.995290 22407 heartbeater.cc:511] Master 127.3.183.254:45911 requested a full tablet report, sending...
I20260929 17:51:46.996249 21947 ts_manager.cc:194] Registered new tserver with Master: b32297b965bc43d9bf67b70e2d00c80a (127.3.183.195:35327)
I20260929 17:51:46.996810 21947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:34433
I20260929 17:51:47.007037 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:51:47.017107 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.008939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.3.183.254:45911)
0929 17:51:47.009094 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.009099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.009264 (+ 165us) server_negotiation.cc:407] Connection header received
0929 17:51:47.009395 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.009398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.009466 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.009542 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:47.010207 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.010841 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.011877 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.012095 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.012171 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.013618 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:47.013621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:47.013623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:47.013649 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:47.015064 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:47.015603 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:47.015606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:47.015608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:47.015690 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:47.016105 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:47.016108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:47.016109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:47.016332 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:47.016463 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.016810 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.016927 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:51:47.020345 21947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:51:47.023324 21947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:51:47.037727 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.031988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.3.183.193:44501)
0929 17:51:47.032726 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.032743 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.032771 (+ 28us) server_negotiation.cc:407] Connection header received
0929 17:51:47.033047 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.033053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.033287 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.033448 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:47.033804 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.034873 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.036375 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.037004 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.037125 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.037268 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.037455 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.037565 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":90,"threads_started":1}
I20260929 17:51:47.037878 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.031820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:44501 (local address 127.0.0.1:39958)
0929 17:51:47.032614 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.032667 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:47.032800 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:47.033475 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:47.033481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:47.033560 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:47.033698 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.033705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.035018 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.035021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.036262 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.036271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.037133 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.037136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.037242 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.037283 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 17:51:47.037401 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 17:51:47.037739 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":90,"threads_started":1}
I20260929 17:51:47.038291 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.032144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:41731 (local address 127.0.0.1:36624)
0929 17:51:47.033479 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.033492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:47.033571 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:47.034241 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:47.034243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:47.034262 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:47.034347 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.034354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.035638 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.035642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.036779 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.036786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.037906 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.037910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.038012 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.038036 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:51:47.038102 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:51:47.038170 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"thread_start_us":94,"threads_started":1}
I20260929 17:51:47.038596 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.032641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.3.183.195:35327)
0929 17:51:47.033155 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.033165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.033551 (+ 386us) server_negotiation.cc:407] Connection header received
0929 17:51:47.033737 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.033742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.033921 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.034076 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:47.034269 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.035074 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.036828 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.037610 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.037703 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.038057 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.038351 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.038446 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":161,"threads_started":1}
I20260929 17:51:47.038609 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.031832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:35327 (local address 127.0.0.1:54920)
0929 17:51:47.033459 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.033472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:47.033610 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:47.034065 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:47.034068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:47.034086 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:47.034169 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.034175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.035218 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.035222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.036700 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.036711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.038120 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.038123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.038211 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.038234 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:51:47.038307 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 17:51:47.038479 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"mutex_wait_us":1,"spinlock_wait_cycles":8576,"thread_start_us":107,"threads_started":1}
I20260929 17:51:47.038681 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.032938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.3.183.194:41731)
0929 17:51:47.033336 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.033349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.033652 (+ 303us) server_negotiation.cc:407] Connection header received
0929 17:51:47.033730 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.033735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.033921 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.034076 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:47.034466 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.035495 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.036877 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.037609 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.037702 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.037831 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.038456 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.038550 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":143,"threads_started":1}
I20260929 17:51:47.041214 22207 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0f8df3ebbd4193b7fb51452e6696dc (DEFAULT_TABLE table=test-table [id=529863705a6e4ccab38d38d137e2676f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:47.041371 22341 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0f8df3ebbd4193b7fb51452e6696dc (DEFAULT_TABLE table=test-table [id=529863705a6e4ccab38d38d137e2676f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:47.041229 22073 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0f8df3ebbd4193b7fb51452e6696dc (DEFAULT_TABLE table=test-table [id=529863705a6e4ccab38d38d137e2676f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:47.042181 22207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0f8df3ebbd4193b7fb51452e6696dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:47.042243 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0f8df3ebbd4193b7fb51452e6696dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:47.042254 22341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0f8df3ebbd4193b7fb51452e6696dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:47.049283 22424 tablet_bootstrap.cc:492] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: Bootstrap starting.
I20260929 17:51:47.050328 22426 tablet_bootstrap.cc:492] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207: Bootstrap starting.
I20260929 17:51:47.051841 22424 tablet_bootstrap.cc:654] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:47.052167 22425 tablet_bootstrap.cc:492] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Bootstrap starting.
I20260929 17:51:47.052237 22426 tablet_bootstrap.cc:654] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:47.052708 22424 log.cc:824] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:47.053402 22426 log.cc:824] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:47.054306 22425 tablet_bootstrap.cc:654] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:47.055004 22424 tablet_bootstrap.cc:492] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: No bootstrap required, opened a new log
I20260929 17:51:47.055223 22424 ts_tablet_manager.cc:1397] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 17:51:47.055231 22425 log.cc:824] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:47.056710 22426 tablet_bootstrap.cc:492] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207: No bootstrap required, opened a new log
I20260929 17:51:47.056903 22426 ts_tablet_manager.cc:1397] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 17:51:47.057221 22425 tablet_bootstrap.cc:492] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: No bootstrap required, opened a new log
I20260929 17:51:47.057390 22425 ts_tablet_manager.cc:1397] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:51:47.058442 22424 raft_consensus.cc:348] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.058683 22424 raft_consensus.cc:374] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:47.058755 22424 raft_consensus.cc:729] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506703c7f26460b87e77815680cb71b, State: Initialized, Role: FOLLOWER
I20260929 17:51:47.059199 22424 consensus_queue.cc:260] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.059890 22273 heartbeater.cc:503] Master 127.3.183.254:45911 was elected leader, sending a full tablet report...
I20260929 17:51:47.060190 22424 ts_tablet_manager.cc:1428] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 17:51:47.060686 22426 raft_consensus.cc:348] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.060884 22426 raft_consensus.cc:374] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:47.060930 22426 raft_consensus.cc:729] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5d7182e08444eca980afb1ed07e207, State: Initialized, Role: FOLLOWER
I20260929 17:51:47.061357 22426 consensus_queue.cc:260] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.061437 22425 raft_consensus.cc:348] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.061720 22425 raft_consensus.cc:374] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:47.061776 22425 raft_consensus.cc:729] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32297b965bc43d9bf67b70e2d00c80a, State: Initialized, Role: FOLLOWER
I20260929 17:51:47.062307 22426 ts_tablet_manager.cc:1428] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:51:47.062359 22425 consensus_queue.cc:260] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.062620 22139 heartbeater.cc:503] Master 127.3.183.254:45911 was elected leader, sending a full tablet report...
I20260929 17:51:47.063215 22425 ts_tablet_manager.cc:1428] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:51:47.063691 22407 heartbeater.cc:503] Master 127.3.183.254:45911 was elected leader, sending a full tablet report...
I20260929 17:51:47.137892 22430 raft_consensus.cc:482] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:47.138109 22430 raft_consensus.cc:504] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.139274 22430 leader_election.cc:288] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5d7182e08444eca980afb1ed07e207 (127.3.183.193:44501), b32297b965bc43d9bf67b70e2d00c80a (127.3.183.195:35327)
I20260929 17:51:47.144017 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.139563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:54991 (local address 127.3.183.193:44501)
0929 17:51:47.139692 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.139697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.139882 (+ 185us) server_negotiation.cc:407] Connection header received
0929 17:51:47.139937 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.139939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.139988 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.140064 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:47.140343 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.141100 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.142669 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.143293 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.143414 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.143467 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.143802 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.143877 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:51:47.144011 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.139704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:43355 (local address 127.3.183.195:35327)
0929 17:51:47.139841 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.139845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.140040 (+ 195us) server_negotiation.cc:407] Connection header received
0929 17:51:47.140159 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.140163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.140200 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.140254 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:47.140506 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.141189 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.142669 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.143293 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.143414 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.143467 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.143809 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.143876 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:51:47.144045 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.139515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:44501 (local address 127.3.183.194:54991)
0929 17:51:47.139689 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.139702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:47.139819 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:47.140093 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:47.140096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:47.140111 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:47.140218 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.140223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.141214 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.141216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.142535 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.142546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.143435 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.143439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.143646 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.143676 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 17:51:47.143757 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:51:47.143862 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":99}
I20260929 17:51:47.144052 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.139546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:35327 (local address 127.3.183.194:43355)
0929 17:51:47.139948 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.139960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:47.140061 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:47.140286 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:47.140289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:47.140306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:47.140410 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.140415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.141284 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.141286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.142535 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.142546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.143435 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.143439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.143632 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.143682 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 17:51:47.143752 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:51:47.143862 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":84,"thread_start_us":138,"threads_started":1}
I20260929 17:51:47.144762 22093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0f8df3ebbd4193b7fb51452e6696dc" candidate_uuid: "a506703c7f26460b87e77815680cb71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5d7182e08444eca980afb1ed07e207" is_pre_election: true
I20260929 17:51:47.144768 22361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0f8df3ebbd4193b7fb51452e6696dc" candidate_uuid: "a506703c7f26460b87e77815680cb71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32297b965bc43d9bf67b70e2d00c80a" is_pre_election: true
I20260929 17:51:47.145099 22361 raft_consensus.cc:2454] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a506703c7f26460b87e77815680cb71b in term 0.
I20260929 17:51:47.145099 22093 raft_consensus.cc:2454] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a506703c7f26460b87e77815680cb71b in term 0.
I20260929 17:51:47.145599 22162 leader_election.cc:302] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a506703c7f26460b87e77815680cb71b, b32297b965bc43d9bf67b70e2d00c80a, ef5d7182e08444eca980afb1ed07e207; no voters:
I20260929 17:51:47.145859 22430 raft_consensus.cc:2781] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:51:47.145941 22430 raft_consensus.cc:482] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:51:47.145988 22430 raft_consensus.cc:3037] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:47.146967 22430 raft_consensus.cc:504] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.147270 22430 leader_election.cc:288] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [CANDIDATE]: Term 1 election: Requested vote from peers ef5d7182e08444eca980afb1ed07e207 (127.3.183.193:44501), b32297b965bc43d9bf67b70e2d00c80a (127.3.183.195:35327)
I20260929 17:51:47.147626 22361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0f8df3ebbd4193b7fb51452e6696dc" candidate_uuid: "a506703c7f26460b87e77815680cb71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32297b965bc43d9bf67b70e2d00c80a"
I20260929 17:51:47.147626 22093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0f8df3ebbd4193b7fb51452e6696dc" candidate_uuid: "a506703c7f26460b87e77815680cb71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5d7182e08444eca980afb1ed07e207"
I20260929 17:51:47.147758 22093 raft_consensus.cc:3037] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:47.147758 22361 raft_consensus.cc:3037] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:47.148985 22093 raft_consensus.cc:2454] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506703c7f26460b87e77815680cb71b in term 1.
I20260929 17:51:47.148985 22361 raft_consensus.cc:2454] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506703c7f26460b87e77815680cb71b in term 1.
I20260929 17:51:47.149329 22161 leader_election.cc:302] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a506703c7f26460b87e77815680cb71b, b32297b965bc43d9bf67b70e2d00c80a, ef5d7182e08444eca980afb1ed07e207; no voters:
I20260929 17:51:47.149562 22430 raft_consensus.cc:2781] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:47.149780 22430 raft_consensus.cc:686] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 1 LEADER]: Becoming Leader. State: Replica: a506703c7f26460b87e77815680cb71b, State: Running, Role: LEADER
I20260929 17:51:47.150246 22430 consensus_queue.cc:237] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } }
I20260929 17:51:47.153594 21948 catalog_manager.cc:5495] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b reported cstate change: term changed from 0 to 1, leader changed from <none> to a506703c7f26460b87e77815680cb71b (127.3.183.194). New cstate: current_term: 1 leader_uuid: "a506703c7f26460b87e77815680cb71b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a506703c7f26460b87e77815680cb71b" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 41731 } health_report { overall_health: HEALTHY } } }
I20260929 17:51:47.191633 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.188587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.3.183.194:41731)
0929 17:51:47.188700 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.188703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.188730 (+ 27us) server_negotiation.cc:407] Connection header received
0929 17:51:47.188796 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.188798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.188851 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.188924 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:47.189133 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.189620 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.190420 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.190633 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.190706 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.190920 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:47.191306 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:51:47.191409 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.191474 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.191508 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:51:47.197365 22361 raft_consensus.cc:1261] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 1 FOLLOWER]: Refusing update from remote peer a506703c7f26460b87e77815680cb71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:47.197365 22093 raft_consensus.cc:1261] T 2a0f8df3ebbd4193b7fb51452e6696dc P ef5d7182e08444eca980afb1ed07e207 [term 1 FOLLOWER]: Refusing update from remote peer a506703c7f26460b87e77815680cb71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:47.198113 22434 consensus_queue.cc:1035] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5d7182e08444eca980afb1ed07e207" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:47.198287 22430 consensus_queue.cc:1035] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32297b965bc43d9bf67b70e2d00c80a" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:47.206107 22439 mvcc.cc:205] Tried to move back new op lower bound from 7334724842273136640 to 7334724842089992192. Current Snapshot: MvccSnapshot[applied={T|T < 7334724842273136640}]
I20260929 17:51:47.216666 21948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:47.216820 21948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:47.219126 21948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63a83f265d1b47d7b05fb375ac77d794: Sending DeleteTablet for 3 replicas of tablet 2a0f8df3ebbd4193b7fb51452e6696dc
I20260929 17:51:47.219887 22073 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0f8df3ebbd4193b7fb51452e6696dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:47 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:39958
I20260929 17:51:47.219933 22207 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0f8df3ebbd4193b7fb51452e6696dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:47 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:36624
I20260929 17:51:47.219980 22341 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0f8df3ebbd4193b7fb51452e6696dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:47 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:54920
I20260929 17:51:47.220393 22449 tablet_replica.cc:330] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: stopping tablet replica
I20260929 17:51:47.220461 22451 tablet_replica.cc:330] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: stopping tablet replica
I20260929 17:51:47.220695 22449 raft_consensus.cc:2229] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:51:47.220870 22451 raft_consensus.cc:2229] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:47.221061 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22010
I20260929 17:51:47.221091 22449 raft_consensus.cc:2258] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:47.221796 22451 raft_consensus.cc:2258] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:47.223186 22451 ts_tablet_manager.cc:1905] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:47.225397 22449 ts_tablet_manager.cc:1905] T 2a0f8df3ebbd4193b7fb51452e6696dc P a506703c7f26460b87e77815680cb71b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:47.226284 22451 ts_tablet_manager.cc:1918] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:47.226413 22451 log.cc:1196] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/2a0f8df3ebbd4193b7fb51452e6696dc
I20260929 17:51:47.226745 22451 ts_tablet_manager.cc:1939] T 2a0f8df3ebbd4193b7fb51452e6696dc P b32297b965bc43d9bf67b70e2d00c80a: Deleting consensus metadata
W20260929 17:51:47.228281 21933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:44501 (error 108)
I20260929 17:51:47.228282 21934 catalog_manager.cc:4841] TS b32297b965bc43d9bf67b70e2d00c80a (127.3.183.195:35327): tablet 2a0f8df3ebbd4193b7fb51452e6696dc (table test-table [id=529863705a6e4ccab38d38d137e2676f]) successfully deleted
I20260929 17:51:47.228507 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22144
I20260929 17:51:47.230091 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.229783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:44501 (local address 127.0.0.1:39968)
0929 17:51:47.229931 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.230005 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:44501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 17:51:47.230248 21933 catalog_manager.cc:4584] TS ef5d7182e08444eca980afb1ed07e207 (127.3.183.193:44501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a0f8df3ebbd4193b7fb51452e6696dc: Network error: Client connection negotiation failed: client connection to 127.3.183.193:44501: connect: Connection refused (error 111)
I20260929 17:51:47.234046 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22278
I20260929 17:51:47.234391 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.234102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:41731 (local address 127.0.0.1:36650)
0929 17:51:47.234264 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.234313 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 17:51:47.238936 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 21916
2026-09-29T17:51:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2194 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T17:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:47Z Disabled control of system clock
I20260929 17:51:47.266883 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41127
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:51:47.373265 22456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:47.373528 22456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:47.373561 22456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:47.377563 22456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:47.377621 22456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:47.377635 22456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:47.377651 22456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:47.377664 22456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:47.382447 22456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:41127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:47.383479 22456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:47.384831 22456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:47.393100 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.393115 22461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.393115 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:47.393167 22456 server_base.cc:1034] running on GCE node
I20260929 17:51:47.393873 22456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:47.394922 22456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:47.396072 22456 hybrid_clock.cc:648] HybridClock initialized: now 1790704307396051 us; error 34 us; skew 500 ppm
I20260929 17:51:47.397861 22456 webserver.cc:460] Webserver started at http://127.3.183.254:33575/ using document root <none> and password file <none>
I20260929 17:51:47.398414 22456 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:47.398479 22456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:47.398676 22456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:47.400727 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "e38aa862b93a4b97887a898aead71be8"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "01ffff0fcb5278eeaaf6d80f9426552c"
server_key_iv: "837a355b252d782edb6da456924c490e"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.401265 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "e38aa862b93a4b97887a898aead71be8"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "01ffff0fcb5278eeaaf6d80f9426552c"
server_key_iv: "837a355b252d782edb6da456924c490e"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.404783 22456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:51:47.407063 22470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:47.408272 22456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:47.408388 22456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "e38aa862b93a4b97887a898aead71be8"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "01ffff0fcb5278eeaaf6d80f9426552c"
server_key_iv: "837a355b252d782edb6da456924c490e"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.408478 22456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:47.418072 22456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:47.418754 22456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:47.418951 22456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:47.425617 22456 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:41127
I20260929 17:51:47.425629 22522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:41127 every 8 connection(s)
I20260929 17:51:47.426548 22456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:47.429545 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:47.432034 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22456
I20260929 17:51:47.432147 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:47.432395 3807 external_mini_cluster.cc:1428] Setting key 2bd5d525e17852c480dcf225be0c7f06
I20260929 17:51:47.435370 22523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8: Bootstrap starting.
I20260929 17:51:47.437589 22523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:47.438271 22523 log.cc:824] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:47.440168 22523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8: No bootstrap required, opened a new log
I20260929 17:51:47.441421 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.433783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.3.183.254:41127)
0929 17:51:47.434421 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.434435 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.434473 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:51:47.435110 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.435127 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.435550 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.435903 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:47.436348 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.437305 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.438198 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.438791 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.438938 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.439724 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:47.439731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:47.439743 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:47.440164 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:47.440218 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:47.440239 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:47.440330 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.440472 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.440734 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
I20260929 17:51:47.443928 22523 raft_consensus.cc:348] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38aa862b93a4b97887a898aead71be8" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41127 } }
I20260929 17:51:47.444142 22523 raft_consensus.cc:374] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:47.444203 22523 raft_consensus.cc:729] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38aa862b93a4b97887a898aead71be8, State: Initialized, Role: FOLLOWER
I20260929 17:51:47.444613 22523 consensus_queue.cc:260] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38aa862b93a4b97887a898aead71be8" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41127 } }
I20260929 17:51:47.444741 22523 raft_consensus.cc:388] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:47.444803 22523 raft_consensus.cc:482] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:47.444900 22523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:47.445849 22523 raft_consensus.cc:504] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38aa862b93a4b97887a898aead71be8" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41127 } }
I20260929 17:51:47.446153 22523 leader_election.cc:302] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e38aa862b93a4b97887a898aead71be8; no voters:
I20260929 17:51:47.446415 22523 leader_election.cc:288] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:47.446547 22528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:47.446733 22528 raft_consensus.cc:686] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [term 1 LEADER]: Becoming Leader. State: Replica: e38aa862b93a4b97887a898aead71be8, State: Running, Role: LEADER
I20260929 17:51:47.447001 22528 consensus_queue.cc:237] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38aa862b93a4b97887a898aead71be8" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41127 } }
I20260929 17:51:47.447553 22523 sys_catalog.cc:564] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:47.448206 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38aa862b93a4b97887a898aead71be8. Latest consensus state: current_term: 1 leader_uuid: "e38aa862b93a4b97887a898aead71be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38aa862b93a4b97887a898aead71be8" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41127 } } }
I20260929 17:51:47.448429 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:47.448518 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e38aa862b93a4b97887a898aead71be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38aa862b93a4b97887a898aead71be8" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41127 } } }
I20260929 17:51:47.448616 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:47.448879 22537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:47.451697 22537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:47.457122 22537 catalog_manager.cc:1347] Generated new cluster ID: f6f9cd4de5be49ac979027fe5c364420
I20260929 17:51:47.457195 22537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:47.465765 22537 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:47.466935 22537 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:47.472582 22537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8: Generated new TSK 0
I20260929 17:51:47.473368 22537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:47.476375 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41127
--builtin_ntp_servers=127.3.183.212:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:51:47.587725 22547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:47.588058 22547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:47.588119 22547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:47.592296 22547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:47.592370 22547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:47.592454 22547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:47.597783 22547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:47.598977 22547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:47.600459 22547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:47.611145 22552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.611150 22553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.611150 22555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:47.611279 22547 server_base.cc:1034] running on GCE node
I20260929 17:51:47.612071 22547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:47.612869 22547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:47.614056 22547 hybrid_clock.cc:648] HybridClock initialized: now 1790704307613992 us; error 82 us; skew 500 ppm
I20260929 17:51:47.616345 22547 webserver.cc:460] Webserver started at http://127.3.183.193:39733/ using document root <none> and password file <none>
I20260929 17:51:47.616961 22547 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:47.617026 22547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:47.617237 22547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:47.619489 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "dc542c64acec42959595f2567a0288e5"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "e9e3697be3aa8bf720c33534a676e5ec"
server_key_iv: "de921d694ae3673c80155f1305496537"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.620085 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "dc542c64acec42959595f2567a0288e5"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "e9e3697be3aa8bf720c33534a676e5ec"
server_key_iv: "de921d694ae3673c80155f1305496537"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.623972 22547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:51:47.626523 22561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:47.627954 22547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:47.628074 22547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "dc542c64acec42959595f2567a0288e5"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "e9e3697be3aa8bf720c33534a676e5ec"
server_key_iv: "de921d694ae3673c80155f1305496537"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.628183 22547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:47.656100 22547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:47.656818 22547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:47.657006 22547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:47.657601 22547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:47.658561 22547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:47.658607 22547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:47.658649 22547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:47.658677 22547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:47.667160 22547 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:35631
I20260929 17:51:47.667202 22674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:35631 every 8 connection(s)
I20260929 17:51:47.668090 22547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:47.673017 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22547
I20260929 17:51:47.673138 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:51:47.673401 3807 external_mini_cluster.cc:1428] Setting key c3c94351c980a1dd0ae91f1e8c5ccfc6
I20260929 17:51:47.675480 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41127
--builtin_ntp_servers=127.3.183.212:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:51:47.678222 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.669981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:55275 (local address 127.3.183.254:41127)
0929 17:51:47.670105 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.670108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.671253 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 17:51:47.672745 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.672749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.672810 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.672891 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:47.673947 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.674571 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.676155 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.676449 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.676551 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.677774 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:47.677777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:47.677780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:47.677794 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:47.677816 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:47.677825 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:47.677888 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.678046 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.678105 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:51:47.679550 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.670399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41127 (local address 127.3.183.193:55275)
0929 17:51:47.671102 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.671142 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:47.672501 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:47.673090 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:47.673099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:47.673213 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:47.673776 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.673794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.674737 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.674741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.676011 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.676023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.676613 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.676617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.676787 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.677475 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:47.677477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:47.677526 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:47.677601 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:47.677902 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:47.677912 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:47.677918 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:51:47.678053 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 17:51:47.678376 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
I20260929 17:51:47.681026 22675 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41127
I20260929 17:51:47.681272 22675 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:47.681865 22675 heartbeater.cc:511] Master 127.3.183.254:41127 requested a full tablet report, sending...
I20260929 17:51:47.683624 22487 ts_manager.cc:194] Registered new tserver with Master: dc542c64acec42959595f2567a0288e5 (127.3.183.193:35631)
I20260929 17:51:47.685350 22487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.193:55275
W20260929 17:51:47.787400 22678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:47.787662 22678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:47.787698 22678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:47.791674 22678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:47.791735 22678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:47.791816 22678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:47.796602 22678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:47.797729 22678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:47.799115 22678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:51:47.808290 22678 server_base.cc:1034] running on GCE node
W20260929 17:51:47.808251 22683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.808251 22684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.808245 22686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:47.808915 22678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:47.809540 22678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:47.810685 22678 hybrid_clock.cc:648] HybridClock initialized: now 1790704307810660 us; error 39 us; skew 500 ppm
I20260929 17:51:47.812506 22678 webserver.cc:460] Webserver started at http://127.3.183.194:43137/ using document root <none> and password file <none>
I20260929 17:51:47.813078 22678 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:47.813141 22678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:47.813300 22678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:47.815333 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "fef169c2b525420ebc07e48a84c16967"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "ae996f52201825d0c7345e9f3d1a69f7"
server_key_iv: "18585c724ba38555c680ca9928333099"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.815889 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "fef169c2b525420ebc07e48a84c16967"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "ae996f52201825d0c7345e9f3d1a69f7"
server_key_iv: "18585c724ba38555c680ca9928333099"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.819490 22678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:51:47.822028 22692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:47.823292 22678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:47.823401 22678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "fef169c2b525420ebc07e48a84c16967"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "ae996f52201825d0c7345e9f3d1a69f7"
server_key_iv: "18585c724ba38555c680ca9928333099"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.823531 22678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:47.833266 22678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:47.833894 22678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:47.834039 22678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:47.834566 22678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:47.835457 22678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:47.835510 22678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:47.835549 22678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:47.835563 22678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:47.844223 22678 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:45573
I20260929 17:51:47.844249 22805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:45573 every 8 connection(s)
I20260929 17:51:47.845141 22678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:47.852310 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22678
I20260929 17:51:47.852437 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:51:47.852698 3807 external_mini_cluster.cc:1428] Setting key 84b345780a320ffaed1e74b5173043dd
I20260929 17:51:47.854305 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.846937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:47755 (local address 127.3.183.254:41127)
0929 17:51:47.847074 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:47.847078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:47.848240 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 17:51:47.849708 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:47.849710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:47.849761 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:47.849823 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:47.850709 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.851247 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.852407 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.852682 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.852775 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.853848 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:47.853852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:47.853855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:47.853870 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:47.853895 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:47.853905 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:47.853973 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:47.854147 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:51:47.854200 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:51:47.854732 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41127
--builtin_ntp_servers=127.3.183.212:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:51:47.855443 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:47.847344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41127 (local address 127.3.183.194:47755)
0929 17:51:47.848100 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:47.848141 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:47.849497 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:47.849992 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:47.849999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:47.850085 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:47.850574 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.850587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.851344 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.851346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.852287 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:47.852296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:47.852825 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:47.852829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:47.853004 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:47.853594 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:47.853596 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:47.853636 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:47.853700 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:47.853996 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:47.854004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:47.854009 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:51:47.854130 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 17:51:47.854414 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
I20260929 17:51:47.856622 22806 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41127
I20260929 17:51:47.856870 22806 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:47.857483 22806 heartbeater.cc:511] Master 127.3.183.254:41127 requested a full tablet report, sending...
I20260929 17:51:47.858580 22487 ts_manager.cc:194] Registered new tserver with Master: fef169c2b525420ebc07e48a84c16967 (127.3.183.194:45573)
I20260929 17:51:47.859171 22487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.194:47755
W20260929 17:51:47.967519 22809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:47.967818 22809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:47.967875 22809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:47.971980 22809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:47.972038 22809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:47.972095 22809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:47.976951 22809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:47.978080 22809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:47.979473 22809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:47.989075 22815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.989171 22814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:47.989286 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:47.989272 22809 server_base.cc:1034] running on GCE node
I20260929 17:51:47.989821 22809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:47.990504 22809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:47.991689 22809 hybrid_clock.cc:648] HybridClock initialized: now 1790704307991673 us; error 28 us; skew 500 ppm
I20260929 17:51:47.993517 22809 webserver.cc:460] Webserver started at http://127.3.183.195:40981/ using document root <none> and password file <none>
I20260929 17:51:47.994071 22809 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:47.994138 22809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:47.994316 22809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:47.996343 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "3a153299c09d4f75827ee05be99f80e4"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "3471aaa34709e8cba7d8ee0dae9593ac"
server_key_iv: "378665349f76bbc89b71c27b1ca363bc"
server_key_version: "encryptionkey@0"
I20260929 17:51:47.996851 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "3a153299c09d4f75827ee05be99f80e4"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "3471aaa34709e8cba7d8ee0dae9593ac"
server_key_iv: "378665349f76bbc89b71c27b1ca363bc"
server_key_version: "encryptionkey@0"
I20260929 17:51:48.000440 22809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:51:48.002823 22823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:48.004094 22809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:48.004200 22809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "3a153299c09d4f75827ee05be99f80e4"
format_stamp: "Formatted at 2026-09-29 17:51:47 on dist-test-slave-wbpt"
server_key: "3471aaa34709e8cba7d8ee0dae9593ac"
server_key_iv: "378665349f76bbc89b71c27b1ca363bc"
server_key_version: "encryptionkey@0"
I20260929 17:51:48.004295 22809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:48.020663 22809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:48.021324 22809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:48.021486 22809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:48.022051 22809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:48.022967 22809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:48.023012 22809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:48.023065 22809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:48.023092 22809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:48.031281 22809 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:43957
I20260929 17:51:48.031320 22936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:43957 every 8 connection(s)
I20260929 17:51:48.032227 22809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:51:48.041215 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.033935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:40109 (local address 127.3.183.254:41127)
0929 17:51:48.034063 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.034067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.035141 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 17:51:48.036626 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.036629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.036680 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.036754 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:48.037722 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.038241 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.039442 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.039672 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.039750 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.040819 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:48.040822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:48.040824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:48.040837 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:48.040856 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:48.040864 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:48.040917 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.041048 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.041100 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:51:48.041793 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22809
I20260929 17:51:48.041893 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:51:48.042091 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.034308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41127 (local address 127.3.183.195:40109)
0929 17:51:48.034992 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:48.035034 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:48.036404 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:48.036954 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:48.036961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:48.037067 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:48.037588 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.037601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.038380 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.038383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.039275 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.039284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.039868 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.039872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.040028 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.040576 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:48.040578 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:48.040618 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:48.040683 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:48.040932 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:48.040938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:48.040943 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:51:48.041046 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 17:51:48.041272 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":118,"threads_started":1}
I20260929 17:51:48.042131 3807 external_mini_cluster.cc:1428] Setting key 1e5b80896d23c2e18df2c42784bfb986
I20260929 17:51:48.043324 22937 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41127
I20260929 17:51:48.043555 22937 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:48.044149 22937 heartbeater.cc:511] Master 127.3.183.254:41127 requested a full tablet report, sending...
I20260929 17:51:48.045094 22486 ts_manager.cc:194] Registered new tserver with Master: 3a153299c09d4f75827ee05be99f80e4 (127.3.183.195:43957)
I20260929 17:51:48.045656 22486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.195:40109
I20260929 17:51:48.056360 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:51:48.062028 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.058371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.3.183.254:41127)
0929 17:51:48.058511 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.058515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.058625 (+ 110us) server_negotiation.cc:407] Connection header received
0929 17:51:48.058730 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.058732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.058771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.058836 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:48.059129 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.059682 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.060531 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.060736 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.060814 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.061607 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:48.061610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:48.061612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:48.061624 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:48.061643 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:48.061651 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:48.061748 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.061831 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.061875 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:51:48.065135 22486 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:51:48.068208 22486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:51:48.082670 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.077349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:45573 (local address 127.0.0.1:51638)
0929 17:51:48.078027 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:48.078040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:48.078123 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:48.078684 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:48.078687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:48.078704 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:48.078793 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.078798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.079762 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.079765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.081153 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.081163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.082087 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.082091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.082192 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.082229 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:51:48.082359 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 17:51:48.082486 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":23,"thread_start_us":105,"threads_started":1}
I20260929 17:51:48.082919 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.076799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.3.183.193:35631)
0929 17:51:48.077309 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.077321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.077804 (+ 483us) server_negotiation.cc:407] Connection header received
0929 17:51:48.077891 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.077897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.078108 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.078236 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:48.078920 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.080081 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.081462 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.082266 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.082376 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.082572 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.082628 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.082741 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":153,"threads_started":1}
I20260929 17:51:48.082711 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.076657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:35631 (local address 127.0.0.1:51416)
0929 17:51:48.077433 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:48.077486 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:48.077664 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:48.078412 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:48.078417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:48.078454 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:48.078698 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.078707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.080206 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.080210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.081343 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.081350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.082381 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.082384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.082471 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.082491 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:51:48.082547 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:51:48.082604 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":100,"spinlock_wait_cycles":6912,"thread_start_us":70,"threads_started":1}
I20260929 17:51:48.083186 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.077008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.3.183.195:43957)
0929 17:51:48.077422 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.077436 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.077723 (+ 287us) server_negotiation.cc:407] Connection header received
0929 17:51:48.077958 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.077963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.078107 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.078234 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:48.078849 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.080081 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.081607 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.082373 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.082463 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.082607 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.082924 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.083035 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":107,"threads_started":1}
I20260929 17:51:48.083425 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.077676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.3.183.194:45573)
0929 17:51:48.078345 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.078356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.078375 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:51:48.078432 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.078436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.078564 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.078691 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:48.078894 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.079653 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.081282 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.081973 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.082075 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.082275 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.083153 (+ 878us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.083254 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":96,"threads_started":1}
I20260929 17:51:48.083611 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.076689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:43957 (local address 127.0.0.1:47658)
0929 17:51:48.077624 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:48.077637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:48.077788 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:48.078412 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:48.078417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:48.078452 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:48.078699 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.078707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.080256 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.080259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.081466 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.081477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.082677 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.082680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.082765 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.082787 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:51:48.083390 (+ 603us) client_negotiation.cc:240] Negotiation successful
0929 17:51:48.083469 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":55,"spinlock_wait_cycles":5888,"thread_start_us":106,"threads_started":1}
I20260929 17:51:48.086328 22740 tablet_service.cc:1467] Processing CreateTablet for tablet de7d797a2c6647af9acc128fcaa956bf (DEFAULT_TABLE table=test-table [id=1388e5866a5d412fa1e8dd0cacdbb99c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:48.086407 22609 tablet_service.cc:1467] Processing CreateTablet for tablet de7d797a2c6647af9acc128fcaa956bf (DEFAULT_TABLE table=test-table [id=1388e5866a5d412fa1e8dd0cacdbb99c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:48.086712 22871 tablet_service.cc:1467] Processing CreateTablet for tablet de7d797a2c6647af9acc128fcaa956bf (DEFAULT_TABLE table=test-table [id=1388e5866a5d412fa1e8dd0cacdbb99c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:48.087344 22740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7d797a2c6647af9acc128fcaa956bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:48.087448 22609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7d797a2c6647af9acc128fcaa956bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:48.087553 22871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7d797a2c6647af9acc128fcaa956bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:48.094957 22955 tablet_bootstrap.cc:492] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Bootstrap starting.
I20260929 17:51:48.096251 22954 tablet_bootstrap.cc:492] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: Bootstrap starting.
I20260929 17:51:48.097085 22956 tablet_bootstrap.cc:492] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Bootstrap starting.
I20260929 17:51:48.097903 22955 tablet_bootstrap.cc:654] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:48.098367 22954 tablet_bootstrap.cc:654] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:48.099129 22956 tablet_bootstrap.cc:654] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:48.099202 22954 log.cc:824] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:48.099921 22956 log.cc:824] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:48.100049 22955 log.cc:824] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:48.101938 22956 tablet_bootstrap.cc:492] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: No bootstrap required, opened a new log
I20260929 17:51:48.101938 22955 tablet_bootstrap.cc:492] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: No bootstrap required, opened a new log
I20260929 17:51:48.101953 22954 tablet_bootstrap.cc:492] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: No bootstrap required, opened a new log
I20260929 17:51:48.102133 22954 ts_tablet_manager.cc:1397] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:51:48.102131 22956 ts_tablet_manager.cc:1397] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:51:48.102131 22955 ts_tablet_manager.cc:1397] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 17:51:48.105731 22954 raft_consensus.cc:348] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.105968 22954 raft_consensus.cc:374] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:48.106037 22954 raft_consensus.cc:729] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc542c64acec42959595f2567a0288e5, State: Initialized, Role: FOLLOWER
I20260929 17:51:48.106252 22956 raft_consensus.cc:348] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.106252 22955 raft_consensus.cc:348] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.106510 22956 raft_consensus.cc:374] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:48.106510 22955 raft_consensus.cc:374] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:48.106565 22955 raft_consensus.cc:729] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef169c2b525420ebc07e48a84c16967, State: Initialized, Role: FOLLOWER
I20260929 17:51:48.106565 22956 raft_consensus.cc:729] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a153299c09d4f75827ee05be99f80e4, State: Initialized, Role: FOLLOWER
I20260929 17:51:48.106493 22954 consensus_queue.cc:260] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.106993 22955 consensus_queue.cc:260] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.106993 22956 consensus_queue.cc:260] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.107633 22937 heartbeater.cc:503] Master 127.3.183.254:41127 was elected leader, sending a full tablet report...
I20260929 17:51:48.107951 22955 ts_tablet_manager.cc:1428] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:51:48.107951 22956 ts_tablet_manager.cc:1428] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:51:48.108258 22806 heartbeater.cc:503] Master 127.3.183.254:41127 was elected leader, sending a full tablet report...
I20260929 17:51:48.107481 22954 ts_tablet_manager.cc:1428] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:51:48.108928 22675 heartbeater.cc:503] Master 127.3.183.254:41127 was elected leader, sending a full tablet report...
I20260929 17:51:48.163511 22962 raft_consensus.cc:482] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:48.163703 22962 raft_consensus.cc:504] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.164835 22962 leader_election.cc:288] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc542c64acec42959595f2567a0288e5 (127.3.183.193:35631), 3a153299c09d4f75827ee05be99f80e4 (127.3.183.195:43957)
I20260929 17:51:48.168696 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.165217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:55891 (local address 127.3.183.195:43957)
0929 17:51:48.165328 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.165333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.165348 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:51:48.165498 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.165501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.165542 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.165607 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:48.165899 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.166554 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.167675 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.168136 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.168230 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.168269 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.168490 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.168563 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:51:48.168709 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.165115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:43957 (local address 127.3.183.194:55891)
0929 17:51:48.165252 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:48.165267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:48.165371 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:48.165632 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:48.165635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:48.165655 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:48.165772 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.165779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.166664 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.166667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.167552 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.167561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.168246 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.168250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.168349 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.168396 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 17:51:48.168464 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:51:48.168562 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":48}
I20260929 17:51:48.168737 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.165240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:44883 (local address 127.3.183.193:35631)
0929 17:51:48.165367 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.165371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.165726 (+ 355us) server_negotiation.cc:407] Connection header received
0929 17:51:48.165847 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.165850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.165903 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.165959 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:48.166231 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.166739 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.167827 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.168235 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.168347 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.168392 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.168566 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.168637 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:51:48.168743 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.165119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:35631 (local address 127.3.183.194:44883)
0929 17:51:48.165647 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:48.165659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:48.165750 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:48.165991 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:48.165994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:48.166010 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:48.166129 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.166136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.166843 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.166846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.167674 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:48.167684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.168353 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.168357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:48.168448 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.168470 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:51:48.168540 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:51:48.168596 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":141,"thread_start_us":183,"threads_started":1}
W20260929 17:51:48.169265 22676 tablet.cc:2369] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:51:48.169378 22891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7d797a2c6647af9acc128fcaa956bf" candidate_uuid: "fef169c2b525420ebc07e48a84c16967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a153299c09d4f75827ee05be99f80e4" is_pre_election: true
I20260929 17:51:48.169390 22629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7d797a2c6647af9acc128fcaa956bf" candidate_uuid: "fef169c2b525420ebc07e48a84c16967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc542c64acec42959595f2567a0288e5" is_pre_election: true
I20260929 17:51:48.169689 22891 raft_consensus.cc:2454] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef169c2b525420ebc07e48a84c16967 in term 0.
I20260929 17:51:48.169690 22629 raft_consensus.cc:2454] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef169c2b525420ebc07e48a84c16967 in term 0.
I20260929 17:51:48.170142 22696 leader_election.cc:302] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc542c64acec42959595f2567a0288e5, fef169c2b525420ebc07e48a84c16967; no voters:
I20260929 17:51:48.170389 22962 raft_consensus.cc:2781] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:51:48.170475 22962 raft_consensus.cc:482] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:51:48.170508 22962 raft_consensus.cc:3037] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:48.171530 22962 raft_consensus.cc:504] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.171908 22962 leader_election.cc:288] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [CANDIDATE]: Term 1 election: Requested vote from peers dc542c64acec42959595f2567a0288e5 (127.3.183.193:35631), 3a153299c09d4f75827ee05be99f80e4 (127.3.183.195:43957)
I20260929 17:51:48.172289 22629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7d797a2c6647af9acc128fcaa956bf" candidate_uuid: "fef169c2b525420ebc07e48a84c16967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc542c64acec42959595f2567a0288e5"
I20260929 17:51:48.172283 22891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7d797a2c6647af9acc128fcaa956bf" candidate_uuid: "fef169c2b525420ebc07e48a84c16967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a153299c09d4f75827ee05be99f80e4"
I20260929 17:51:48.172405 22891 raft_consensus.cc:3037] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:48.172410 22629 raft_consensus.cc:3037] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:48.173686 22629 raft_consensus.cc:2454] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef169c2b525420ebc07e48a84c16967 in term 1.
I20260929 17:51:48.173682 22891 raft_consensus.cc:2454] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef169c2b525420ebc07e48a84c16967 in term 1.
I20260929 17:51:48.174021 22693 leader_election.cc:302] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a153299c09d4f75827ee05be99f80e4, dc542c64acec42959595f2567a0288e5, fef169c2b525420ebc07e48a84c16967; no voters:
I20260929 17:51:48.174203 22962 raft_consensus.cc:2781] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:48.174434 22962 raft_consensus.cc:686] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 1 LEADER]: Becoming Leader. State: Replica: fef169c2b525420ebc07e48a84c16967, State: Running, Role: LEADER
I20260929 17:51:48.174757 22962 consensus_queue.cc:237] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } }
I20260929 17:51:48.178171 22485 catalog_manager.cc:5495] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef169c2b525420ebc07e48a84c16967 (127.3.183.194). New cstate: current_term: 1 leader_uuid: "fef169c2b525420ebc07e48a84c16967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef169c2b525420ebc07e48a84c16967" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 45573 } health_report { overall_health: HEALTHY } } }
I20260929 17:51:48.212533 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.209101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.3.183.194:45573)
0929 17:51:48.209222 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:48.209227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:48.209244 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:51:48.209315 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:48.209318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:48.209370 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:48.209488 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:48.209700 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.210191 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.211055 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:48.211263 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:48.211347 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:48.211659 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:48.212122 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:51:48.212250 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:48.212347 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 17:51:48.212382 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:51:48.218230 22891 raft_consensus.cc:1261] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 1 FOLLOWER]: Refusing update from remote peer fef169c2b525420ebc07e48a84c16967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:48.218230 22629 raft_consensus.cc:1261] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 1 FOLLOWER]: Refusing update from remote peer fef169c2b525420ebc07e48a84c16967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:48.218933 22964 consensus_queue.cc:1035] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a153299c09d4f75827ee05be99f80e4" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 43957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:48.219120 22962 consensus_queue.cc:1035] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc542c64acec42959595f2567a0288e5" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:48.228853 22969 mvcc.cc:205] Tried to move back new op lower bound from 7334724846454534144 to 7334724846286196736. Current Snapshot: MvccSnapshot[applied={T|T < 7334724846454534144}]
I20260929 17:51:48.235875 22485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:48.236059 22485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:48.238505 22485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e38aa862b93a4b97887a898aead71be8: Sending DeleteTablet for 3 replicas of tablet de7d797a2c6647af9acc128fcaa956bf
I20260929 17:51:48.239230 22609 tablet_service.cc:1514] Processing DeleteTablet for tablet de7d797a2c6647af9acc128fcaa956bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:48 UTC) from {username='slave'} at 127.0.0.1:51416
I20260929 17:51:48.239344 22871 tablet_service.cc:1514] Processing DeleteTablet for tablet de7d797a2c6647af9acc128fcaa956bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:48 UTC) from {username='slave'} at 127.0.0.1:47658
I20260929 17:51:48.239459 22740 tablet_service.cc:1514] Processing DeleteTablet for tablet de7d797a2c6647af9acc128fcaa956bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:48 UTC) from {username='slave'} at 127.0.0.1:51638
I20260929 17:51:48.239748 22977 tablet_replica.cc:330] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5: stopping tablet replica
I20260929 17:51:48.240130 22976 tablet_replica.cc:330] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: stopping tablet replica
I20260929 17:51:48.240432 22977 raft_consensus.cc:2229] T de7d797a2c6647af9acc128fcaa956bf P dc542c64acec42959595f2567a0288e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:48.240432 22976 raft_consensus.cc:2229] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:48.240517 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22547
I20260929 17:51:48.242075 22978 tablet_replica.cc:330] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: stopping tablet replica
I20260929 17:51:48.242354 22978 raft_consensus.cc:2229] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:51:48.242689 22978 raft_consensus.cc:2258] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:48.243057 22976 raft_consensus.cc:2258] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:48.244385 22978 ts_tablet_manager.cc:1905] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:48.247045 22978 ts_tablet_manager.cc:1918] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:48.247192 22978 log.cc:1196] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/de7d797a2c6647af9acc128fcaa956bf
I20260929 17:51:48.247601 22978 ts_tablet_manager.cc:1939] T de7d797a2c6647af9acc128fcaa956bf P fef169c2b525420ebc07e48a84c16967: Deleting consensus metadata
W20260929 17:51:48.247864 22474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:35631 (error 108)
I20260929 17:51:48.248014 22976 ts_tablet_manager.cc:1905] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:48.248756 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22678
I20260929 17:51:48.248922 22471 catalog_manager.cc:4841] TS fef169c2b525420ebc07e48a84c16967 (127.3.183.194:45573): tablet de7d797a2c6647af9acc128fcaa956bf (table test-table [id=1388e5866a5d412fa1e8dd0cacdbb99c]) successfully deleted
I20260929 17:51:48.249907 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:48.249623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:35631 (local address 127.0.0.1:51428)
0929 17:51:48.249752 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:48.249821 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 17:51:48.250383 22474 catalog_manager.cc:4584] TS dc542c64acec42959595f2567a0288e5 (127.3.183.193:35631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de7d797a2c6647af9acc128fcaa956bf: Network error: Client connection negotiation failed: client connection to 127.3.183.193:35631: connect: Connection refused (error 111)
I20260929 17:51:48.251024 22976 ts_tablet_manager.cc:1918] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:48.251107 22976 log.cc:1196] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/de7d797a2c6647af9acc128fcaa956bf
I20260929 17:51:48.251349 22976 ts_tablet_manager.cc:1939] T de7d797a2c6647af9acc128fcaa956bf P 3a153299c09d4f75827ee05be99f80e4: Deleting consensus metadata
I20260929 17:51:48.252583 22471 catalog_manager.cc:4841] TS 3a153299c09d4f75827ee05be99f80e4 (127.3.183.195:43957): tablet de7d797a2c6647af9acc128fcaa956bf (table test-table [id=1388e5866a5d412fa1e8dd0cacdbb99c]) successfully deleted
I20260929 17:51:48.254629 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22809
I20260929 17:51:48.259290 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22456
2026-09-29T17:51:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1019 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:48 dist-test-slave-wbpt krb5kdc[22982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:48 dist-test-slave-wbpt krb5kdc[22982](info): set up 2 sockets
Sep 29 17:51:48 dist-test-slave-wbpt krb5kdc[22982](info): commencing operation
krb5kdc: starting...
W20260929 17:51:49.231596 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:49 dist-test-slave-wbpt krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:51:49.351254 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41655
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:41655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:49.459369 22995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:49.459661 22995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:49.459717 22995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:49.463771 22995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:49.463864 22995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:49.463891 22995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:49.463912 22995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:49.463932 22995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:49.469252 22995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:41655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:41655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:49.470319 22995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:49.471346 22995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:49.479820 23003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:49.479852 22995 server_base.cc:1034] running on GCE node
W20260929 17:51:49.479852 23001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:49.480633 23000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:49.481272 22995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:49.482585 22995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:49.483757 22995 hybrid_clock.cc:648] HybridClock initialized: now 1790704309483724 us; error 47 us; skew 500 ppm
Sep 29 17:51:49 dist-test-slave-wbpt krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:49.486554 22995 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:51:49.487448 22995 webserver.cc:460] Webserver started at http://127.3.183.254:35803/ using document root <none> and password file <none>
I20260929 17:51:49.488021 22995 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:49.488082 22995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:49.488251 22995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:49.490262 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "6a4f3f376f2b465abfc473b079697ffe"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "d8698618b2f3030c799ee52c521f44ac"
server_key_iv: "8fb0ebbdefc1510c7678ad75d3bbbc37"
server_key_version: "encryptionkey@0"
I20260929 17:51:49.490840 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "6a4f3f376f2b465abfc473b079697ffe"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "d8698618b2f3030c799ee52c521f44ac"
server_key_iv: "8fb0ebbdefc1510c7678ad75d3bbbc37"
server_key_version: "encryptionkey@0"
I20260929 17:51:49.494517 22995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:51:49.496896 23010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:49.498114 22995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:51:49.498214 22995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "6a4f3f376f2b465abfc473b079697ffe"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "d8698618b2f3030c799ee52c521f44ac"
server_key_iv: "8fb0ebbdefc1510c7678ad75d3bbbc37"
server_key_version: "encryptionkey@0"
I20260929 17:51:49.498301 22995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:49.526525 22995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:49.529606 22995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:49.529771 22995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:49.536439 22995 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:41655
I20260929 17:51:49.536487 23062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:41655 every 8 connection(s)
I20260929 17:51:49.537386 22995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:49.537568 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 22995
I20260929 17:51:49.537693 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:49.537986 3807 external_mini_cluster.cc:1428] Setting key f243ac3298d9292653b4cf0678356e86
I20260929 17:51:49.540387 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 17:51:49 dist-test-slave-wbpt krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:49.546890 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe: Bootstrap starting.
I20260929 17:51:49.549422 23063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:49.550304 23063 log.cc:824] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:49.552454 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe: No bootstrap required, opened a new log
I20260929 17:51:49.552454 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:49.539354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.3.183.254:41655)
0929 17:51:49.539756 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:49.539772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:49.539838 (+ 66us) server_negotiation.cc:407] Connection header received
0929 17:51:49.540501 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:49.540525 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:49.540854 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:49.541136 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:49.542008 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:49.543012 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:49.544197 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:49.544668 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:49.544786 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:49.547611 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:49.547616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:49.547631 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:49.547660 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:49.549392 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:49.549901 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:49.549906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:49.549911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:49.550028 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:49.550330 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:49.550333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:49.550334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:49.550874 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:49.551068 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:49.551343 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 17:51:49.551666 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260929 17:51:49.556051 23063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4f3f376f2b465abfc473b079697ffe" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41655 } }
I20260929 17:51:49.556281 23063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:49.556341 23063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4f3f376f2b465abfc473b079697ffe, State: Initialized, Role: FOLLOWER
I20260929 17:51:49.556763 23063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4f3f376f2b465abfc473b079697ffe" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41655 } }
I20260929 17:51:49.556922 23063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:49.556991 23063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:49.557116 23063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:49.558100 23063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4f3f376f2b465abfc473b079697ffe" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41655 } }
I20260929 17:51:49.558405 23063 leader_election.cc:302] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a4f3f376f2b465abfc473b079697ffe; no voters:
I20260929 17:51:49.558673 23063 leader_election.cc:288] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:49.558799 23068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:49.559038 23068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [term 1 LEADER]: Becoming Leader. State: Replica: 6a4f3f376f2b465abfc473b079697ffe, State: Running, Role: LEADER
I20260929 17:51:49.559321 23068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4f3f376f2b465abfc473b079697ffe" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41655 } }
I20260929 17:51:49.559849 23063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:49.560977 23069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a4f3f376f2b465abfc473b079697ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4f3f376f2b465abfc473b079697ffe" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41655 } } }
I20260929 17:51:49.561100 23069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:49.561359 23070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a4f3f376f2b465abfc473b079697ffe. Latest consensus state: current_term: 1 leader_uuid: "6a4f3f376f2b465abfc473b079697ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4f3f376f2b465abfc473b079697ffe" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 41655 } } }
I20260929 17:51:49.561444 23070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:49.561842 23077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:49.564584 23077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:49.570075 23077 catalog_manager.cc:1347] Generated new cluster ID: f08cc2133e454c1c9597ae54cfa8bd15
I20260929 17:51:49.570153 23077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:49.583374 23077 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:49.584762 23077 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:49.598810 23077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe: Generated new TSK 0
I20260929 17:51:49.599745 23077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.183.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
Entry for principal kudu/127.3.183.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab.
I20260929 17:51:49.628809 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41655
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:49.741823 23089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:49.742118 23089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:49.742161 23089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:49.746397 23089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:49.746467 23089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:49.746537 23089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:49.751930 23089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.193.keytab
--principal=kudu/127.3.183.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:49.753038 23089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:49.754096 23089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:49.763973 23095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:49.764045 23097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:49.764045 23094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:49.764273 23089 server_base.cc:1034] running on GCE node
I20260929 17:51:49.764714 23089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:49.765398 23089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:49.766628 23089 hybrid_clock.cc:648] HybridClock initialized: now 1790704309766599 us; error 39 us; skew 500 ppm
Sep 29 17:51:49 dist-test-slave-wbpt krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:49.769426 23089 init.cc:377] Logged in from keytab as kudu/127.3.183.193@KRBTEST.COM (short username kudu)
I20260929 17:51:49.770367 23089 webserver.cc:460] Webserver started at http://127.3.183.193:36681/ using document root <none> and password file <none>
I20260929 17:51:49.770903 23089 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:49.770968 23089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:49.771135 23089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:49.773217 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "76c20f65a56a4ff1bc70ff6afc4f1657"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "f29b5e9057476b35521aec050c0ee102"
server_key_iv: "2dfd377508da3ccead1c27abac4a70c9"
server_key_version: "encryptionkey@0"
I20260929 17:51:49.773757 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "76c20f65a56a4ff1bc70ff6afc4f1657"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "f29b5e9057476b35521aec050c0ee102"
server_key_iv: "2dfd377508da3ccead1c27abac4a70c9"
server_key_version: "encryptionkey@0"
I20260929 17:51:49.778139 23089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:51:49.780666 23104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:49.781958 23089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:49.782063 23089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "76c20f65a56a4ff1bc70ff6afc4f1657"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "f29b5e9057476b35521aec050c0ee102"
server_key_iv: "2dfd377508da3ccead1c27abac4a70c9"
server_key_version: "encryptionkey@0"
I20260929 17:51:49.782177 23089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:49.807156 23089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:49.810139 23089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:49.810315 23089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:49.810916 23089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:49.811771 23089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:49.811851 23089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:49.811898 23089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:49.811926 23089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:49.820326 23089 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:39003
I20260929 17:51:49.820356 23217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:39003 every 8 connection(s)
I20260929 17:51:49.821215 23089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:49.825398 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 23089
I20260929 17:51:49.825544 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:51:49.825821 3807 external_mini_cluster.cc:1428] Setting key d8b174ba7d6d411f7830c62f2624cb28
Sep 29 17:51:49 dist-test-slave-wbpt krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.193@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.183.194@KRBTEST.COM; defaulting to no policy
I20260929 17:51:49.835728 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:49.822965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:58775 (local address 127.3.183.254:41655)
0929 17:51:49.823123 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:49.823128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:49.824009 (+ 881us) server_negotiation.cc:407] Connection header received
0929 17:51:49.825277 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:49.825281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:49.825343 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:49.825449 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:49.826806 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:49.827480 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:49.829135 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:49.829370 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:49.829462 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:49.831984 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:49.831989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:49.831993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:49.832026 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:49.833609 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:49.834126 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:49.834130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:49.834132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:49.834219 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:49.834557 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:49.834560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:49.834562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:49.834803 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:49.834915 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:49.835474 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 17:51:49.835597 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:51:49.836665 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:49.823388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41655 (local address 127.3.183.193:58775)
0929 17:51:49.823866 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:49.823910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:49.825060 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:49.825671 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:49.825679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:49.826080 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:49.826663 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:49.826679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:49.827620 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:49.827624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:49.829006 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:49.829018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:49.829526 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:49.829530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:49.829703 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:49.830282 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:49.830309 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:49.831765 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:49.833734 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:49.833740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:49.833754 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:49.834019 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:49.834306 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:49.834308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:49.834310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:49.834436 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:49.834937 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:49.834942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:49.835204 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 17:51:49.835388 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 17:51:49.835699 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260929 17:51:49.837930 23218 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.194@KRBTEST.COM" created.
I20260929 17:51:49.838171 23218 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:49.838708 23218 heartbeater.cc:511] Master 127.3.183.254:41655 requested a full tablet report, sending...
I20260929 17:51:49.840549 23027 ts_manager.cc:194] Registered new tserver with Master: 76c20f65a56a4ff1bc70ff6afc4f1657 (127.3.183.193:39003)
I20260929 17:51:49.841655 23027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.193@KRBTEST.COM'} at 127.3.183.193:58775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
Entry for principal kudu/127.3.183.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab.
I20260929 17:51:49.855189 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41655
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:49.966251 23223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:49.966550 23223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:49.966609 23223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:49.970772 23223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:49.970852 23223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:49.970934 23223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:49.976400 23223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.194.keytab
--principal=kudu/127.3.183.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:49.977561 23223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:49.978609 23223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:49.988437 23229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:49.988437 23231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:49.988668 23228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:49.988922 23223 server_base.cc:1034] running on GCE node
I20260929 17:51:49.989377 23223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:49.990041 23223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:49.991233 23223 hybrid_clock.cc:648] HybridClock initialized: now 1790704309991205 us; error 38 us; skew 500 ppm
Sep 29 17:51:49 dist-test-slave-wbpt krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:49.994513 23223 init.cc:377] Logged in from keytab as kudu/127.3.183.194@KRBTEST.COM (short username kudu)
I20260929 17:51:49.995597 23223 webserver.cc:460] Webserver started at http://127.3.183.194:33815/ using document root <none> and password file <none>
I20260929 17:51:49.996204 23223 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:49.996275 23223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:49.996513 23223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:49.998687 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "ab9e58ef70ca922d95fd4fe7170baf37"
server_key_iv: "a569e4c2d6ca636c42d6fa52efaeb66b"
server_key_version: "encryptionkey@0"
I20260929 17:51:49.999302 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "ab9e58ef70ca922d95fd4fe7170baf37"
server_key_iv: "a569e4c2d6ca636c42d6fa52efaeb66b"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.003278 23223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:51:50.006202 23238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.007504 23223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:51:50.007625 23223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999"
format_stamp: "Formatted at 2026-09-29 17:51:49 on dist-test-slave-wbpt"
server_key: "ab9e58ef70ca922d95fd4fe7170baf37"
server_key_iv: "a569e4c2d6ca636c42d6fa52efaeb66b"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.007733 23223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:50.019120 23223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:50.022101 23223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:50.022292 23223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:50.022894 23223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:50.023732 23223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:50.023808 23223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.023874 23223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:50.023904 23223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.032481 23223 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:39223
I20260929 17:51:50.032501 23351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:39223 every 8 connection(s)
I20260929 17:51:50.033430 23223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:50.041875 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 23223
I20260929 17:51:50.042008 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:51:50.042361 3807 external_mini_cluster.cc:1428] Setting key 81b472c55ae0b807bfd765cd3d21851d
Sep 29 17:51:50 dist-test-slave-wbpt krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.194@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:50.048879 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.035489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:46713 (local address 127.3.183.254:41655)
0929 17:51:50.035656 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.035660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.036647 (+ 987us) server_negotiation.cc:407] Connection header received
0929 17:51:50.037973 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.037977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.038035 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.038110 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:50.039450 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.040026 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.041183 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.041405 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.041495 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.044393 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:50.044396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:50.044399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:50.044425 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:50.046188 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:50.046805 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:50.046810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:50.046811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:50.046952 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:50.047353 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:50.047356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:50.047358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:50.047608 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:50.047718 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.048551 (+ 833us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.048701 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 17:51:50.049794 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.035998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41655 (local address 127.3.183.194:46713)
0929 17:51:50.036478 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.036530 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.037744 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.038363 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.038371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.038763 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:50.039289 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.039310 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.040147 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.040150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.041061 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.041071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.041558 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.041562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.041748 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.042387 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:50.042403 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:50.044209 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:50.046338 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:50.046345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:50.046364 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:50.046673 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:50.047068 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:50.047070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:50.047072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:50.047257 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:50.047741 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:50.047748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:50.048195 (+ 447us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.048485 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.048823 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260929 17:51:50.051141 23352 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41655
I20260929 17:51:50.051453 23352 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:50.052294 23352 heartbeater.cc:511] Master 127.3.183.254:41655 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.183.195@KRBTEST.COM; defaulting to no policy
I20260929 17:51:50.053716 23027 ts_manager.cc:194] Registered new tserver with Master: ae18ca32fd15465d9a5cb6ec3e9e9999 (127.3.183.194:39223)
I20260929 17:51:50.054399 23027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.194@KRBTEST.COM'} at 127.3.183.194:46713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
Entry for principal kudu/127.3.183.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab.
I20260929 17:51:50.076021 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41655
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:50.188201 23357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:50.188447 23357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:50.188513 23357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:50.192521 23357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:50.192577 23357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:50.192652 23357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:50.197818 23357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.195.keytab
--principal=kudu/127.3.183.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:41655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:50.198891 23357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:50.199918 23357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:50.209666 23365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:50.209782 23357 server_base.cc:1034] running on GCE node
W20260929 17:51:50.209671 23363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:50.209674 23362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:50.210323 23357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:50.211030 23357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:50.212188 23357 hybrid_clock.cc:648] HybridClock initialized: now 1790704310212170 us; error 32 us; skew 500 ppm
Sep 29 17:51:50 dist-test-slave-wbpt krb5kdc[22982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:50.214818 23357 init.cc:377] Logged in from keytab as kudu/127.3.183.195@KRBTEST.COM (short username kudu)
I20260929 17:51:50.215724 23357 webserver.cc:460] Webserver started at http://127.3.183.195:42357/ using document root <none> and password file <none>
I20260929 17:51:50.216315 23357 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:50.216383 23357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:50.216591 23357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:50.218706 23357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "b7cd27294a664d79afb2b0e7935a141c"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "2e5ec1e0de89e07342ea890f5d205a3b"
server_key_iv: "ab1935ccc2153faa5c436cc8867f79a6"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.219244 23357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "b7cd27294a664d79afb2b0e7935a141c"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "2e5ec1e0de89e07342ea890f5d205a3b"
server_key_iv: "ab1935ccc2153faa5c436cc8867f79a6"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.223117 23357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:51:50.225706 23372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.226980 23357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:50.227108 23357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "b7cd27294a664d79afb2b0e7935a141c"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "2e5ec1e0de89e07342ea890f5d205a3b"
server_key_iv: "ab1935ccc2153faa5c436cc8867f79a6"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.227211 23357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:50.268363 23357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:50.271323 23357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:50.271543 23357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:50.272150 23357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:50.272970 23357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:50.273016 23357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.273068 23357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:50.273092 23357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.281584 23357 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:32887
I20260929 17:51:50.281630 23485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:32887 every 8 connection(s)
I20260929 17:51:50.282533 23357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:51:50.282754 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 23357
I20260929 17:51:50.282884 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:51:50.283244 3807 external_mini_cluster.cc:1428] Setting key 0474ebcaf4a3ca5968c0a325770a7011
Sep 29 17:51:50 dist-test-slave-wbpt krb5kdc[22982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.195@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:51:50.295696 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.284498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:53481 (local address 127.3.183.254:41655)
0929 17:51:50.284625 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.284629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.285451 (+ 822us) server_negotiation.cc:407] Connection header received
0929 17:51:50.286743 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.286746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.286800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.286895 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:50.288089 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.288669 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.289760 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.289984 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.290068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.292441 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:50.292444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:50.292446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:50.292478 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:50.293938 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:50.294396 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:50.294399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:50.294400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:50.294518 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:50.294776 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:50.294778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:50.294779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:50.294945 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:50.295018 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.295425 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.295552 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:51:50.296597 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.284875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:41655 (local address 127.3.183.195:53481)
0929 17:51:50.285307 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.285368 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.286553 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.287091 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.287098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.287444 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:50.287963 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.287979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.288780 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.288782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.289648 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.289657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.290118 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.290122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.290275 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.290790 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:51:50.290805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:50.292261 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:50.294043 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:50.294048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:50.294060 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:50.294293 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:50.294594 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:51:50.294596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:51:50.294597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:51:50.294714 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:51:50.295017 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:50.295022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:50.295241 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.295397 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.295659 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20260929 17:51:50.297657 23486 heartbeater.cc:347] Connected to a master server at 127.3.183.254:41655
I20260929 17:51:50.297854 23486 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:50.298280 23486 heartbeater.cc:511] Master 127.3.183.254:41655 requested a full tablet report, sending...
I20260929 17:51:50.299292 23027 ts_manager.cc:194] Registered new tserver with Master: b7cd27294a664d79afb2b0e7935a141c (127.3.183.195:32887)
I20260929 17:51:50.299921 23027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.183.195@KRBTEST.COM'} at 127.3.183.195:53481
I20260929 17:51:50.308853 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:51:50.319499 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.310694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.3.183.254:41655)
0929 17:51:50.310837 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.310842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.310977 (+ 135us) server_negotiation.cc:407] Connection header received
0929 17:51:50.311072 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.311075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.311115 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.311177 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:50.312089 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.312860 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.314066 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.314424 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.314543 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.315934 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:50.315938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:50.315942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:50.315971 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:50.317566 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:50.318061 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:50.318064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:50.318067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:50.318164 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:50.318508 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:50.318511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:50.318512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:50.318793 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:50.318923 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.319181 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.319311 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:51:50.322609 23027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:51:50.325748 23027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:51:50.339850 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.334011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39003 (local address 127.0.0.1:34084)
0929 17:51:50.335063 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.335074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.335167 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.335672 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.335678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.335711 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:50.335946 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.335953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.337393 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.337396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.338503 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.338512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.339378 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.339380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.339484 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.339516 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.339600 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.339688 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":62,"spinlock_wait_cycles":5888,"thread_start_us":102,"threads_started":1}
I20260929 17:51:50.339922 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.334658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.3.183.193:39003)
0929 17:51:50.335017 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.335032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.335160 (+ 128us) server_negotiation.cc:407] Connection header received
0929 17:51:50.335268 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.335273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.335518 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.335673 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:50.336077 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.337274 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.338616 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.339276 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.339391 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.339538 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.339644 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.339744 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":74,"threads_started":1}
I20260929 17:51:50.340440 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.334210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:32887 (local address 127.0.0.1:54528)
0929 17:51:50.335941 (+ 1731us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.335958 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.336040 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.336386 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.336389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.336405 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:50.336497 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.336503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.337682 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.337685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.338861 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.338872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.340077 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.340081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.340166 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.340190 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.340254 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.340318 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1674,"mutex_wait_us":41,"thread_start_us":78,"threads_started":1}
I20260929 17:51:50.340859 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.334646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.3.183.195:32887)
0929 17:51:50.335112 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.335125 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.336037 (+ 912us) server_negotiation.cc:407] Connection header received
0929 17:51:50.336136 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.336140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.336274 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.336399 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:50.336579 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.337378 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.338980 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.339747 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.339878 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.340014 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.340641 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.340732 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":135,"threads_started":1}
I20260929 17:51:50.342460 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.333900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39223 (local address 127.0.0.1:39056)
0929 17:51:50.334650 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.334700 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.334816 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.335672 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.335678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.335712 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:50.335946 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.335953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.337545 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.337548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.338861 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.338872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.342071 (+ 3199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.342075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.342163 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.342186 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.342256 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.342324 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":114,"spinlock_wait_cycles":5248,"thread_start_us":81,"threads_started":1}
I20260929 17:51:50.342963 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.334057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.3.183.194:39223)
0929 17:51:50.335312 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.335322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.335341 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:51:50.335404 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.335408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.335547 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.335673 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:50.336077 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.337417 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.341171 (+ 3754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.341947 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.342052 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.342178 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.342730 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.342828 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1165,"thread_start_us":104,"threads_started":1}
I20260929 17:51:50.343101 23152 tablet_service.cc:1467] Processing CreateTablet for tablet 3f192233478a4a5e917e2e894a71831f (DEFAULT_TABLE table=test-table [id=fbe2749f924c415f80dc1f573631e9b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:50.343395 23420 tablet_service.cc:1467] Processing CreateTablet for tablet 3f192233478a4a5e917e2e894a71831f (DEFAULT_TABLE table=test-table [id=fbe2749f924c415f80dc1f573631e9b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:50.344103 23152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f192233478a4a5e917e2e894a71831f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:50.344280 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f192233478a4a5e917e2e894a71831f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:50.349845 23503 tablet_bootstrap.cc:492] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657: Bootstrap starting.
I20260929 17:51:50.350560 23286 tablet_service.cc:1467] Processing CreateTablet for tablet 3f192233478a4a5e917e2e894a71831f (DEFAULT_TABLE table=test-table [id=fbe2749f924c415f80dc1f573631e9b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:50.351701 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f192233478a4a5e917e2e894a71831f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:50.351701 23504 tablet_bootstrap.cc:492] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Bootstrap starting.
I20260929 17:51:50.352156 23503 tablet_bootstrap.cc:654] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:50.352849 23503 log.cc:824] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:50.353848 23504 tablet_bootstrap.cc:654] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:50.354954 23503 tablet_bootstrap.cc:492] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657: No bootstrap required, opened a new log
I20260929 17:51:50.355073 23504 log.cc:824] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:50.355142 23503 ts_tablet_manager.cc:1397] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:51:50.357141 23504 tablet_bootstrap.cc:492] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: No bootstrap required, opened a new log
I20260929 17:51:50.357326 23504 ts_tablet_manager.cc:1397] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:51:50.358672 23503 raft_consensus.cc:348] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.358934 23503 raft_consensus.cc:374] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:50.358989 23503 raft_consensus.cc:729] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c20f65a56a4ff1bc70ff6afc4f1657, State: Initialized, Role: FOLLOWER
I20260929 17:51:50.359416 23503 consensus_queue.cc:260] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.359997 23507 tablet_bootstrap.cc:492] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Bootstrap starting.
I20260929 17:51:50.360412 23503 ts_tablet_manager.cc:1428] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:51:50.360520 23218 heartbeater.cc:503] Master 127.3.183.254:41655 was elected leader, sending a full tablet report...
I20260929 17:51:50.361372 23504 raft_consensus.cc:348] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.361682 23504 raft_consensus.cc:374] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:50.361744 23504 raft_consensus.cc:729] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cd27294a664d79afb2b0e7935a141c, State: Initialized, Role: FOLLOWER
I20260929 17:51:50.362160 23504 consensus_queue.cc:260] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.362311 23507 tablet_bootstrap.cc:654] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:50.362813 23486 heartbeater.cc:503] Master 127.3.183.254:41655 was elected leader, sending a full tablet report...
I20260929 17:51:50.363133 23504 ts_tablet_manager.cc:1428] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:51:50.363495 23507 log.cc:824] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:50.365178 23507 tablet_bootstrap.cc:492] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: No bootstrap required, opened a new log
I20260929 17:51:50.365346 23507 ts_tablet_manager.cc:1397] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 17:51:50.368211 23507 raft_consensus.cc:348] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.368428 23507 raft_consensus.cc:374] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:50.368481 23507 raft_consensus.cc:729] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae18ca32fd15465d9a5cb6ec3e9e9999, State: Initialized, Role: FOLLOWER
I20260929 17:51:50.368842 23507 consensus_queue.cc:260] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.369392 23352 heartbeater.cc:503] Master 127.3.183.254:41655 was elected leader, sending a full tablet report...
I20260929 17:51:50.369618 23507 ts_tablet_manager.cc:1428] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260929 17:51:50.383708 23508 raft_consensus.cc:482] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:50.383948 23508 raft_consensus.cc:504] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.385161 23508 leader_election.cc:288] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae18ca32fd15465d9a5cb6ec3e9e9999 (127.3.183.194:39223), b7cd27294a664d79afb2b0e7935a141c (127.3.183.195:32887)
I20260929 17:51:50.389341 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.385464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:47081 (local address 127.3.183.194:39223)
0929 17:51:50.385598 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.385603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.386009 (+ 406us) server_negotiation.cc:407] Connection header received
0929 17:51:50.386174 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.386176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.386224 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.386300 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:50.386767 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.387269 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.388455 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.388843 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.388948 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.389003 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.389156 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.389220 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:51:50.389357 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.385788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:55493 (local address 127.3.183.195:32887)
0929 17:51:50.385906 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.385910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.386039 (+ 129us) server_negotiation.cc:407] Connection header received
0929 17:51:50.386164 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.386173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.386220 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.386298 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:50.386716 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.387276 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.388445 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.388863 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.388965 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.389012 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.389180 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.389241 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:51:50.389398 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.385372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:32887 (local address 127.3.183.193:55493)
0929 17:51:50.385918 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.385932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.386068 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.386314 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.386317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.386341 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:50.386603 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.386611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.387380 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.387384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.388328 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.388338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.388977 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.388981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.389073 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.389090 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.389152 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.389228 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":71,"spinlock_wait_cycles":20736,"thread_start_us":107,"threads_started":1}
I20260929 17:51:50.389405 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.385352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:39223 (local address 127.3.183.193:47081)
0929 17:51:50.385925 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.385937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.386068 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.386314 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.386317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.386338 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:50.386603 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.386611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.387373 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.387376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.388337 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.388347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.388951 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.388955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.389047 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.389072 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.389135 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.389209 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":122,"spinlock_wait_cycles":25600,"thread_start_us":124,"threads_started":1}
I20260929 17:51:50.390033 23440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f192233478a4a5e917e2e894a71831f" candidate_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd27294a664d79afb2b0e7935a141c" is_pre_election: true
I20260929 17:51:50.390146 23306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f192233478a4a5e917e2e894a71831f" candidate_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" is_pre_election: true
I20260929 17:51:50.390318 23440 raft_consensus.cc:2454] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c20f65a56a4ff1bc70ff6afc4f1657 in term 0.
I20260929 17:51:50.390396 23306 raft_consensus.cc:2454] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c20f65a56a4ff1bc70ff6afc4f1657 in term 0.
I20260929 17:51:50.390734 23106 leader_election.cc:302] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c20f65a56a4ff1bc70ff6afc4f1657, b7cd27294a664d79afb2b0e7935a141c; no voters:
I20260929 17:51:50.390974 23508 raft_consensus.cc:2781] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:51:50.391052 23508 raft_consensus.cc:482] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:51:50.391095 23508 raft_consensus.cc:3037] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:50.392179 23508 raft_consensus.cc:504] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.392504 23508 leader_election.cc:288] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [CANDIDATE]: Term 1 election: Requested vote from peers ae18ca32fd15465d9a5cb6ec3e9e9999 (127.3.183.194:39223), b7cd27294a664d79afb2b0e7935a141c (127.3.183.195:32887)
I20260929 17:51:50.392874 23306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f192233478a4a5e917e2e894a71831f" candidate_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999"
I20260929 17:51:50.392900 23440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f192233478a4a5e917e2e894a71831f" candidate_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd27294a664d79afb2b0e7935a141c"
I20260929 17:51:50.392990 23306 raft_consensus.cc:3037] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:50.393011 23440 raft_consensus.cc:3037] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:50.393965 23306 raft_consensus.cc:2454] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c20f65a56a4ff1bc70ff6afc4f1657 in term 1.
I20260929 17:51:50.393996 23440 raft_consensus.cc:2454] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c20f65a56a4ff1bc70ff6afc4f1657 in term 1.
I20260929 17:51:50.394272 23105 leader_election.cc:302] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c20f65a56a4ff1bc70ff6afc4f1657, ae18ca32fd15465d9a5cb6ec3e9e9999; no voters:
I20260929 17:51:50.394485 23508 raft_consensus.cc:2781] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:50.394692 23508 raft_consensus.cc:686] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 1 LEADER]: Becoming Leader. State: Replica: 76c20f65a56a4ff1bc70ff6afc4f1657, State: Running, Role: LEADER
I20260929 17:51:50.395017 23508 consensus_queue.cc:237] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } }
I20260929 17:51:50.398147 23026 catalog_manager.cc:5495] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c20f65a56a4ff1bc70ff6afc4f1657 (127.3.183.193). New cstate: current_term: 1 leader_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c20f65a56a4ff1bc70ff6afc4f1657" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 39003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 } health_report { overall_health: UNKNOWN } } }
I20260929 17:51:50.422381 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.418930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.3.183.193:39003)
0929 17:51:50.419075 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.419079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.419092 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:51:50.419161 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.419163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.419214 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.419300 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:50.419572 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.420104 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.421034 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.421285 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.421385 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.421605 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:50.422015 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:51:50.422099 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.422185 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.422236 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 17:51:50.428150 23306 raft_consensus.cc:1261] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 1 FOLLOWER]: Refusing update from remote peer 76c20f65a56a4ff1bc70ff6afc4f1657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:50.428150 23440 raft_consensus.cc:1261] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 1 FOLLOWER]: Refusing update from remote peer 76c20f65a56a4ff1bc70ff6afc4f1657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:50.428879 23508 consensus_queue.cc:1035] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae18ca32fd15465d9a5cb6ec3e9e9999" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 39223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:50.429075 23514 consensus_queue.cc:1035] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cd27294a664d79afb2b0e7935a141c" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:50.437532 23517 mvcc.cc:205] Tried to move back new op lower bound from 7334724855506444288 to 7334724855380688896. Current Snapshot: MvccSnapshot[applied={T|T < 7334724855506444288}]
I20260929 17:51:50.446586 23026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:50.446735 23026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:50.449193 23026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a4f3f376f2b465abfc473b079697ffe: Sending DeleteTablet for 3 replicas of tablet 3f192233478a4a5e917e2e894a71831f
I20260929 17:51:50.449998 23152 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f192233478a4a5e917e2e894a71831f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:34084
I20260929 17:51:50.450001 23286 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f192233478a4a5e917e2e894a71831f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:39056
I20260929 17:51:50.450510 23528 tablet_replica.cc:330] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657: stopping tablet replica
I20260929 17:51:50.450505 23420 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f192233478a4a5e917e2e894a71831f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:50 UTC) from {username='kudu', principal='kudu/127.3.183.254@KRBTEST.COM'} at 127.0.0.1:54528
I20260929 17:51:50.450830 23528 raft_consensus.cc:2229] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:51:50.450946 23530 tablet_replica.cc:330] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: stopping tablet replica
I20260929 17:51:50.451232 23528 raft_consensus.cc:2258] T 3f192233478a4a5e917e2e894a71831f P 76c20f65a56a4ff1bc70ff6afc4f1657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:50.451223 23530 raft_consensus.cc:2229] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:50.451443 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 23089
I20260929 17:51:50.451654 23530 raft_consensus.cc:2258] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:50.453413 23529 tablet_replica.cc:330] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: stopping tablet replica
I20260929 17:51:50.453684 23529 raft_consensus.cc:2229] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:50.453970 23529 raft_consensus.cc:2258] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:50.454214 23530 ts_tablet_manager.cc:1905] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:50.455269 23529 ts_tablet_manager.cc:1905] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:50.457357 23530 ts_tablet_manager.cc:1918] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:50.457399 23529 ts_tablet_manager.cc:1918] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:50.457455 23530 log.cc:1196] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/3f192233478a4a5e917e2e894a71831f
I20260929 17:51:50.457479 23529 log.cc:1196] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704058755242-3807-0/minicluster-data/ts-1/wal/wals/3f192233478a4a5e917e2e894a71831f
I20260929 17:51:50.457749 23529 ts_tablet_manager.cc:1939] T 3f192233478a4a5e917e2e894a71831f P ae18ca32fd15465d9a5cb6ec3e9e9999: Deleting consensus metadata
I20260929 17:51:50.457758 23530 ts_tablet_manager.cc:1939] T 3f192233478a4a5e917e2e894a71831f P b7cd27294a664d79afb2b0e7935a141c: Deleting consensus metadata
I20260929 17:51:50.458910 23011 catalog_manager.cc:4841] TS ae18ca32fd15465d9a5cb6ec3e9e9999 (127.3.183.194:39223): tablet 3f192233478a4a5e917e2e894a71831f (table test-table [id=fbe2749f924c415f80dc1f573631e9b1]) successfully deleted
I20260929 17:51:50.458972 23012 catalog_manager.cc:4841] TS b7cd27294a664d79afb2b0e7935a141c (127.3.183.195:32887): tablet 3f192233478a4a5e917e2e894a71831f (table test-table [id=fbe2749f924c415f80dc1f573631e9b1]) successfully deleted
W20260929 17:51:50.459178 23013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:39003 (error 108)
I20260929 17:51:50.459921 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 23223
I20260929 17:51:50.460139 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.459870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:39003 (local address 127.0.0.1:34094)
0929 17:51:50.459993 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.460063 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 17:51:50.460572 23013 catalog_manager.cc:4584] TS 76c20f65a56a4ff1bc70ff6afc4f1657 (127.3.183.193:39003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f192233478a4a5e917e2e894a71831f: Network error: Client connection negotiation failed: client connection to 127.3.183.193:39003: connect: Connection refused (error 111)
I20260929 17:51:50.465193 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 23357
I20260929 17:51:50.469604 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 22995
2026-09-29T17:51:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2211 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T17:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:50Z Disabled control of system clock
I20260929 17:51:50.498459 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34323
--webserver_interface=127.3.183.254
--webserver_port=0
--builtin_ntp_servers=127.3.183.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:51:50.608764 23535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:50.609064 23535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:50.609100 23535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:50.613143 23535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:50.613231 23535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:50.613255 23535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:50.613274 23535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:50.613293 23535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:50.618297 23535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:34323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:50.619415 23535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:50.620877 23535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:50.629249 23541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:50.629370 23535 server_base.cc:1034] running on GCE node
W20260929 17:51:50.629263 23540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:50.629246 23543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:50.630230 23535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:50.631362 23535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:50.632525 23535 hybrid_clock.cc:648] HybridClock initialized: now 1790704310632501 us; error 40 us; skew 500 ppm
I20260929 17:51:50.634311 23535 webserver.cc:460] Webserver started at http://127.3.183.254:41449/ using document root <none> and password file <none>
I20260929 17:51:50.634874 23535 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:50.634943 23535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:50.635133 23535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:50.637235 23535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "183769e88b2647049eaeda0936c9499a"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "547e5b62949fb6afde5134963edfece8"
server_key_iv: "dd6d45e342cd6a34623c558bf0903e59"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.637774 23535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "183769e88b2647049eaeda0936c9499a"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "547e5b62949fb6afde5134963edfece8"
server_key_iv: "dd6d45e342cd6a34623c558bf0903e59"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.641335 23535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 17:51:50.643625 23549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.645015 23535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:50.645138 23535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "183769e88b2647049eaeda0936c9499a"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "547e5b62949fb6afde5134963edfece8"
server_key_iv: "dd6d45e342cd6a34623c558bf0903e59"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.645237 23535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:50.659053 23535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:50.659811 23535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:50.659996 23535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:50.667073 23535 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:34323
I20260929 17:51:50.667085 23601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:34323 every 8 connection(s)
I20260929 17:51:50.668078 23535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:50.671180 23602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:50.674417 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 23535
I20260929 17:51:50.674546 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:50.674804 3807 external_mini_cluster.cc:1428] Setting key 7e547148beb59c85f47b1ebc14f5c6c2
I20260929 17:51:50.677758 23602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a: Bootstrap starting.
I20260929 17:51:50.680090 23602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:50.680840 23602 log.cc:824] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:50.682919 23602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a: No bootstrap required, opened a new log
I20260929 17:51:50.685286 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.676324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.3.183.254:34323)
0929 17:51:50.677090 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.677111 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.677152 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:51:50.677883 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.677908 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.678368 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.678699 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:50.679178 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.680245 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.681240 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.681768 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.681945 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.682932 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:50.682937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:50.682950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:50.683479 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:50.683580 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:50.683611 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:50.683714 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.684007 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.684519 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
I20260929 17:51:50.686867 23602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183769e88b2647049eaeda0936c9499a" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34323 } }
I20260929 17:51:50.687108 23602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:50.687152 23602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183769e88b2647049eaeda0936c9499a, State: Initialized, Role: FOLLOWER
I20260929 17:51:50.687599 23602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183769e88b2647049eaeda0936c9499a" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34323 } }
I20260929 17:51:50.687728 23602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:50.687806 23602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:50.687912 23602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:50.688850 23602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183769e88b2647049eaeda0936c9499a" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34323 } }
I20260929 17:51:50.689181 23602 leader_election.cc:302] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 183769e88b2647049eaeda0936c9499a; no voters:
I20260929 17:51:50.689473 23602 leader_election.cc:288] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:50.689558 23607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:50.689761 23607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [term 1 LEADER]: Becoming Leader. State: Replica: 183769e88b2647049eaeda0936c9499a, State: Running, Role: LEADER
I20260929 17:51:50.690058 23607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183769e88b2647049eaeda0936c9499a" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34323 } }
I20260929 17:51:50.690606 23602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:50.691671 23608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "183769e88b2647049eaeda0936c9499a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183769e88b2647049eaeda0936c9499a" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34323 } } }
I20260929 17:51:50.691831 23608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:50.692219 23609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 183769e88b2647049eaeda0936c9499a. Latest consensus state: current_term: 1 leader_uuid: "183769e88b2647049eaeda0936c9499a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183769e88b2647049eaeda0936c9499a" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34323 } } }
I20260929 17:51:50.692315 23609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:50.692589 23616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:50.695395 23616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:50.701076 23616 catalog_manager.cc:1347] Generated new cluster ID: 4fa660958e5647c9a07fd5e0929bdab3
I20260929 17:51:50.701162 23616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:50.722487 23616 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:50.723613 23616 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:50.731348 23616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a: Generated new TSK 0
I20260929 17:51:50.732278 23616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:50.736344 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.193:0
--local_ip_for_outbound_sockets=127.3.183.193
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34323
--builtin_ntp_servers=127.3.183.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:51:50.846815 23626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:50.847081 23626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:50.847136 23626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:50.851238 23626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:50.851316 23626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:50.851392 23626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.193
I20260929 17:51:50.856140 23626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.183.193
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.193
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:50.857177 23626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:50.858542 23626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:50.868240 23632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:50.868263 23631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:50.868402 23626 server_base.cc:1034] running on GCE node
W20260929 17:51:50.868240 23634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:50.868880 23626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:50.869510 23626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:50.870671 23626 hybrid_clock.cc:648] HybridClock initialized: now 1790704310870652 us; error 29 us; skew 500 ppm
I20260929 17:51:50.872543 23626 webserver.cc:460] Webserver started at http://127.3.183.193:36717/ using document root <none> and password file <none>
I20260929 17:51:50.873139 23626 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:50.873206 23626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:50.873396 23626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:50.875466 23626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data/instance:
uuid: "a5b1cf4ba9f2440a841fa989f6dea233"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "60caecc64c7ed78f96d750e1acb61f56"
server_key_iv: "066c78d2bbe089232f07c43d2d3554b8"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.876051 23626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance:
uuid: "a5b1cf4ba9f2440a841fa989f6dea233"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "60caecc64c7ed78f96d750e1acb61f56"
server_key_iv: "066c78d2bbe089232f07c43d2d3554b8"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.879908 23626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 17:51:50.882443 23640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.883937 23626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:50.884055 23626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/wal
uuid: "a5b1cf4ba9f2440a841fa989f6dea233"
format_stamp: "Formatted at 2026-09-29 17:51:50 on dist-test-slave-wbpt"
server_key: "60caecc64c7ed78f96d750e1acb61f56"
server_key_iv: "066c78d2bbe089232f07c43d2d3554b8"
server_key_version: "encryptionkey@0"
I20260929 17:51:50.884152 23626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:50.908931 23626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:50.909662 23626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:50.909840 23626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:50.910434 23626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:50.911413 23626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:50.911461 23626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.911515 23626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:50.911540 23626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:50.919744 23626 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.193:41773
I20260929 17:51:50.919802 23753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.193:41773 every 8 connection(s)
I20260929 17:51:50.920651 23626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/data/info.pb
I20260929 17:51:50.922574 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 23626
I20260929 17:51:50.922704 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-0/wal/instance
I20260929 17:51:50.923012 3807 external_mini_cluster.cc:1428] Setting key 4ae0c6ec6654fda5bcfd7acb869c357c
I20260929 17:51:50.924984 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.194:0
--local_ip_for_outbound_sockets=127.3.183.194
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34323
--builtin_ntp_servers=127.3.183.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:51:50.929973 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.922351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.193:58579 (local address 127.3.183.254:34323)
0929 17:51:50.922481 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:50.922484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:50.923810 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 17:51:50.925273 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:50.925276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:50.925335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:50.925408 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:50.926324 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.926862 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.928036 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.928269 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.928359 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.929513 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:50.929516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:50.929518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:50.929531 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:50.929550 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:50.929558 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:50.929616 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:50.929766 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 17:51:50.929828 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:51:50.930838 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:50.922886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34323 (local address 127.3.183.193:58579)
0929 17:51:50.923656 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:50.923698 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:50.925034 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:50.925594 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:50.925601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:50.925687 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:50.926183 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.926195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.926992 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.926995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.927910 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:50.927920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:50.928420 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:50.928425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:50.928593 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:50.929224 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:50.929226 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:50.929268 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:50.929335 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:50.929629 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:50.929640 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:50.929647 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:51:50.929757 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 17:51:50.930067 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260929 17:51:50.932116 23754 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34323
I20260929 17:51:50.932415 23754 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:50.933122 23754 heartbeater.cc:511] Master 127.3.183.254:34323 requested a full tablet report, sending...
I20260929 17:51:50.934942 23566 ts_manager.cc:194] Registered new tserver with Master: a5b1cf4ba9f2440a841fa989f6dea233 (127.3.183.193:41773)
I20260929 17:51:50.936228 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.193:58579
W20260929 17:51:51.037395 23757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:51.037688 23757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:51.037735 23757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:51.041862 23757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:51.041929 23757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:51.041987 23757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.194
I20260929 17:51:51.046893 23757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.183.194
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.194
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:51.048039 23757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:51.049435 23757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:51.059664 23765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:51.059554 23762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:51.059590 23757 server_base.cc:1034] running on GCE node
W20260929 17:51:51.059597 23763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:51.060202 23757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:51.060976 23757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:51.062160 23757 hybrid_clock.cc:648] HybridClock initialized: now 1790704311062132 us; error 46 us; skew 500 ppm
I20260929 17:51:51.064112 23757 webserver.cc:460] Webserver started at http://127.3.183.194:41861/ using document root <none> and password file <none>
I20260929 17:51:51.064679 23757 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:51.064750 23757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:51.064971 23757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:51.067070 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data/instance:
uuid: "c3ef2c747e31413190e7fc4cfe83b7fc"
format_stamp: "Formatted at 2026-09-29 17:51:51 on dist-test-slave-wbpt"
server_key: "511726e34eb0c498f887b254f10e14bb"
server_key_iv: "37090122fdf5d3904b00afd66996763e"
server_key_version: "encryptionkey@0"
I20260929 17:51:51.067633 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance:
uuid: "c3ef2c747e31413190e7fc4cfe83b7fc"
format_stamp: "Formatted at 2026-09-29 17:51:51 on dist-test-slave-wbpt"
server_key: "511726e34eb0c498f887b254f10e14bb"
server_key_iv: "37090122fdf5d3904b00afd66996763e"
server_key_version: "encryptionkey@0"
I20260929 17:51:51.071494 23757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:51:51.074209 23771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:51.075557 23757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:51:51.075681 23757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/wal
uuid: "c3ef2c747e31413190e7fc4cfe83b7fc"
format_stamp: "Formatted at 2026-09-29 17:51:51 on dist-test-slave-wbpt"
server_key: "511726e34eb0c498f887b254f10e14bb"
server_key_iv: "37090122fdf5d3904b00afd66996763e"
server_key_version: "encryptionkey@0"
I20260929 17:51:51.075824 23757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:51.095317 23757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:51.096099 23757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:51.096285 23757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:51.096872 23757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:51.097860 23757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:51.097915 23757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:51.097970 23757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:51.097996 23757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:51.106962 23757 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.194:37905
I20260929 17:51:51.107008 23884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.194:37905 every 8 connection(s)
I20260929 17:51:51.107928 23757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/data/info.pb
I20260929 17:51:51.111502 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 23757
I20260929 17:51:51.111620 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-1/wal/instance
I20260929 17:51:51.111901 3807 external_mini_cluster.cc:1428] Setting key 7b3d0cc9649aeeb2d2ad987edb243e91
I20260929 17:51:51.113885 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.183.195:0
--local_ip_for_outbound_sockets=127.3.183.195
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34323
--builtin_ntp_servers=127.3.183.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:51:51.117250 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.109750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:41509 (local address 127.3.183.254:34323)
0929 17:51:51.109891 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.109897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.111061 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 17:51:51.112574 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.112577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.112630 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.112710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:51.113783 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.114345 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.115450 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.115661 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.115744 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.116795 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:51.116798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:51.116800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:51.116812 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:51.116831 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:51.116839 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:51.116896 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.117080 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.117139 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:51:51.118387 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.110228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34323 (local address 127.3.183.194:41509)
0929 17:51:51.110906 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.110949 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:51.112357 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:51.112907 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:51.112916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:51.113032 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:51.113651 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.113665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.114459 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.114463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.115337 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.115346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.115825 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.115830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.115990 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.116539 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:51.116541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:51.116589 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:51.116654 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:51.116904 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:51.116913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:51.116919 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:51:51.117400 (+ 481us) client_negotiation.cc:240] Negotiation successful
0929 17:51:51.117648 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260929 17:51:51.119489 23885 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34323
I20260929 17:51:51.119717 23885 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:51.120285 23885 heartbeater.cc:511] Master 127.3.183.254:34323 requested a full tablet report, sending...
I20260929 17:51:51.121331 23566 ts_manager.cc:194] Registered new tserver with Master: c3ef2c747e31413190e7fc4cfe83b7fc (127.3.183.194:37905)
I20260929 17:51:51.122108 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.194:41509
W20260929 17:51:51.224669 23888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:51.224958 23888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:51.224998 23888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:51.229148 23888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:51.229213 23888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:51.229274 23888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.183.195
I20260929 17:51:51.234170 23888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.183.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.183.195
--webserver_port=0
--tserver_master_addrs=127.3.183.254:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.183.195
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:51.235255 23888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:51.236668 23888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:51.245945 23894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:51.245981 23896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:51.245999 23893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:51.246050 23888 server_base.cc:1034] running on GCE node
I20260929 17:51:51.246604 23888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:51.247270 23888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:51.248458 23888 hybrid_clock.cc:648] HybridClock initialized: now 1790704311248435 us; error 34 us; skew 500 ppm
I20260929 17:51:51.250370 23888 webserver.cc:460] Webserver started at http://127.3.183.195:35473/ using document root <none> and password file <none>
I20260929 17:51:51.250950 23888 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:51.251019 23888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:51.251205 23888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:51.253342 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data/instance:
uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8"
format_stamp: "Formatted at 2026-09-29 17:51:51 on dist-test-slave-wbpt"
server_key: "aa3e3621aea41ef443980e2e7913a70c"
server_key_iv: "ece5f213c0b2f35972d52ac12b2d1ad7"
server_key_version: "encryptionkey@0"
I20260929 17:51:51.253902 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance:
uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8"
format_stamp: "Formatted at 2026-09-29 17:51:51 on dist-test-slave-wbpt"
server_key: "aa3e3621aea41ef443980e2e7913a70c"
server_key_iv: "ece5f213c0b2f35972d52ac12b2d1ad7"
server_key_version: "encryptionkey@0"
I20260929 17:51:51.257680 23888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:51:51.260186 23902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:51.261459 23888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:51.261571 23888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal
uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8"
format_stamp: "Formatted at 2026-09-29 17:51:51 on dist-test-slave-wbpt"
server_key: "aa3e3621aea41ef443980e2e7913a70c"
server_key_iv: "ece5f213c0b2f35972d52ac12b2d1ad7"
server_key_version: "encryptionkey@0"
I20260929 17:51:51.261672 23888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:51.278257 23888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:51.279055 23888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:51.279240 23888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:51.279877 23888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:51:51.280876 23888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:51:51.280933 23888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:51.280974 23888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:51:51.280992 23888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:51.289233 23888 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.195:42951
I20260929 17:51:51.289268 24015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.195:42951 every 8 connection(s)
I20260929 17:51:51.290227 23888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/data/info.pb
I20260929 17:51:51.299256 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.292067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.195:35315 (local address 127.3.183.254:34323)
0929 17:51:51.292192 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.292196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.293331 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 17:51:51.294770 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.294773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.294824 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.294898 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:51.295840 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.296397 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.297513 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.297731 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.297807 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.298839 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:51.298842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:51.298845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:51.298858 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:51.298878 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:51.298886 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:51.298940 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.299082 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.299135 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:51:51.300128 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 23888
I20260929 17:51:51.300148 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.292554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:34323 (local address 127.3.183.195:35315)
0929 17:51:51.293200 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.293242 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:51.294552 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:51.295053 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:51.295060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:51.295145 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:51:51.295659 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.295672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.296500 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.296503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.297380 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.297389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.297846 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.297849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.297993 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.298588 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:51:51.298590 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:51:51.298632 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:51:51.298699 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:51:51.298961 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:51:51.298970 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:51:51.298976 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:51:51.299076 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 17:51:51.299307 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260929 17:51:51.300230 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal/instance
I20260929 17:51:51.300590 3807 external_mini_cluster.cc:1428] Setting key 80141c0b848e34de69b2240453398d26
I20260929 17:51:51.301334 24016 heartbeater.cc:347] Connected to a master server at 127.3.183.254:34323
I20260929 17:51:51.301604 24016 heartbeater.cc:464] Registering TS with master...
I20260929 17:51:51.302168 24016 heartbeater.cc:511] Master 127.3.183.254:34323 requested a full tablet report, sending...
I20260929 17:51:51.303107 23566 ts_manager.cc:194] Registered new tserver with Master: 4e07d9a5aa66450f82ca9d5afeca7ff8 (127.3.183.195:42951)
I20260929 17:51:51.303653 23566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.183.195:35315
I20260929 17:51:51.314515 3807 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:51:51.320425 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.316585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.3.183.254:34323)
0929 17:51:51.316711 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.316715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.316912 (+ 197us) server_negotiation.cc:407] Connection header received
0929 17:51:51.316991 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.316993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.317037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.317126 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:51.317509 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.318049 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.318900 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.319120 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.319204 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.320007 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:51.320010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:51.320012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:51.320024 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:51.320042 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:51.320050 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:51.320112 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.320211 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.320270 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:51:51.323305 23566 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:51:51.326270 23566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:51:51.340917 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.334618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.3.183.193:41773)
0929 17:51:51.335398 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.335409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.335434 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:51:51.335515 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.335520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.335731 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.335931 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:51.336396 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.337645 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.339219 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.340090 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.340197 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.340395 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.340615 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.340726 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":155,"threads_started":1}
I20260929 17:51:51.340907 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.335188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:42951 (local address 127.0.0.1:52174)
0929 17:51:51.335453 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.335465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:51.335541 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:51.336571 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:51.336574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:51.336590 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:51.336645 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.336649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.337606 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.337610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.339184 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.339196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.340202 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.340206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.340406 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.340458 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 17:51:51.340574 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 17:51:51.340749 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":260,"thread_start_us":85,"threads_started":1}
I20260929 17:51:51.340907 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.334453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:41773 (local address 127.0.0.1:55024)
0929 17:51:51.334795 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.334841 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:51.334960 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:51.336110 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:51.336116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:51.336138 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:51.336271 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.336279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.337795 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.337798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.339101 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.339112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.340202 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.340206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.340395 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.340458 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 17:51:51.340574 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 17:51:51.340735 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":145,"thread_start_us":92,"threads_started":1}
I20260929 17:51:51.341403 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.335372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.3.183.195:42951)
0929 17:51:51.336001 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.336013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.336038 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:51:51.336110 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.336115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.336285 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.336429 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:51.336745 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.337493 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.339314 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.340090 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.340196 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.340404 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.341140 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.341243 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":103,"threads_started":1}
I20260929 17:51:51.341894 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.334698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:37905 (local address 127.0.0.1:34640)
0929 17:51:51.335058 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.335071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:51.335160 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:51.336330 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:51.336333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:51.336352 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:51.336458 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.336464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.337778 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.337781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.339184 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.339196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.341476 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.341480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.341586 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.341609 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:51:51.341659 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 17:51:51.341765 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":88,"thread_start_us":89,"threads_started":1}
I20260929 17:51:51.342700 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.335547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.3.183.194:37905)
0929 17:51:51.335884 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.335895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.335926 (+ 31us) server_negotiation.cc:407] Connection header received
0929 17:51:51.335996 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.336000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.336190 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.336355 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:51.336544 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.337657 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.340467 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.341321 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.342222 (+ 901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.342458 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.342502 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.342581 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
I20260929 17:51:51.344393 23688 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa84a06c664478088d89d868d04badf (DEFAULT_TABLE table=test-table [id=f1b61f1e16914f73974c46f90b765cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:51.344753 23950 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa84a06c664478088d89d868d04badf (DEFAULT_TABLE table=test-table [id=f1b61f1e16914f73974c46f90b765cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:51.345620 23688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fa84a06c664478088d89d868d04badf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:51.345727 23950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fa84a06c664478088d89d868d04badf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:51.345465 23819 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa84a06c664478088d89d868d04badf (DEFAULT_TABLE table=test-table [id=f1b61f1e16914f73974c46f90b765cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:51:51.346406 23819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fa84a06c664478088d89d868d04badf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:51.352855 24033 tablet_bootstrap.cc:492] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Bootstrap starting.
I20260929 17:51:51.353538 24034 tablet_bootstrap.cc:492] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233: Bootstrap starting.
I20260929 17:51:51.354336 24035 tablet_bootstrap.cc:492] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: Bootstrap starting.
I20260929 17:51:51.355300 24033 tablet_bootstrap.cc:654] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:51.355669 24034 tablet_bootstrap.cc:654] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:51.356060 24035 tablet_bootstrap.cc:654] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:51.356251 24033 log.cc:824] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:51.356457 24034 log.cc:824] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:51.356809 24035 log.cc:824] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:51.358405 24034 tablet_bootstrap.cc:492] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233: No bootstrap required, opened a new log
I20260929 17:51:51.358405 24033 tablet_bootstrap.cc:492] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: No bootstrap required, opened a new log
I20260929 17:51:51.358462 24035 tablet_bootstrap.cc:492] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: No bootstrap required, opened a new log
I20260929 17:51:51.358606 24033 ts_tablet_manager.cc:1397] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:51:51.358606 24034 ts_tablet_manager.cc:1397] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:51:51.358639 24035 ts_tablet_manager.cc:1397] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 17:51:51.361892 24035 raft_consensus.cc:348] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.362126 24035 raft_consensus.cc:374] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:51.362197 24035 raft_consensus.cc:729] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ef2c747e31413190e7fc4cfe83b7fc, State: Initialized, Role: FOLLOWER
I20260929 17:51:51.362717 24035 consensus_queue.cc:260] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.362744 24033 raft_consensus.cc:348] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.362744 24034 raft_consensus.cc:348] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.363019 24034 raft_consensus.cc:374] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:51.363001 24039 raft_consensus.cc:482] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:51:51.363116 24033 raft_consensus.cc:374] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:51.363114 24034 raft_consensus.cc:729] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b1cf4ba9f2440a841fa989f6dea233, State: Initialized, Role: FOLLOWER
I20260929 17:51:51.363131 24039 raft_consensus.cc:504] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.363217 24033 raft_consensus.cc:729] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e07d9a5aa66450f82ca9d5afeca7ff8, State: Initialized, Role: FOLLOWER
I20260929 17:51:51.363607 24033 consensus_queue.cc:260] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.363606 24034 consensus_queue.cc:260] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.364627 24034 ts_tablet_manager.cc:1428] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:51:51.364627 24033 ts_tablet_manager.cc:1428] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:51:51.364777 24035 ts_tablet_manager.cc:1428] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:51:51.364996 23754 heartbeater.cc:503] Master 127.3.183.254:34323 was elected leader, sending a full tablet report...
I20260929 17:51:51.365137 23885 heartbeater.cc:503] Master 127.3.183.254:34323 was elected leader, sending a full tablet report...
I20260929 17:51:51.365208 24016 heartbeater.cc:503] Master 127.3.183.254:34323 was elected leader, sending a full tablet report...
I20260929 17:51:51.366852 24039 leader_election.cc:288] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b1cf4ba9f2440a841fa989f6dea233 (127.3.183.193:41773), 4e07d9a5aa66450f82ca9d5afeca7ff8 (127.3.183.195:42951)
I20260929 17:51:51.369786 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.365030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:59713 (local address 127.3.183.193:41773)
0929 17:51:51.365124 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.365128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.365144 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:51:51.365188 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.365191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.365231 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.365305 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:51.365760 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.366433 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.368577 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.369122 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.369231 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.369278 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.369596 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.369670 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 17:51:51.370301 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.364530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:41773 (local address 127.3.183.194:59713)
0929 17:51:51.364775 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.364790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:51.364874 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:51.365418 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:51.365421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:51.365439 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:51.365576 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.365583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.367263 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.367267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.368431 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.368442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.369336 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.369340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.369437 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.369474 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:51:51.369539 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:51:51.370186 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":555}
I20260929 17:51:51.370829 23708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fa84a06c664478088d89d868d04badf" candidate_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" is_pre_election: true
I20260929 17:51:51.371116 23708 raft_consensus.cc:2454] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ef2c747e31413190e7fc4cfe83b7fc in term 0.
I20260929 17:51:51.371585 23775 leader_election.cc:302] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b1cf4ba9f2440a841fa989f6dea233, c3ef2c747e31413190e7fc4cfe83b7fc; no voters:
I20260929 17:51:51.371708 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.366647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.195:42951 (local address 127.3.183.194:34537)
0929 17:51:51.367352 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.367365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:51:51.367490 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:51:51.367745 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:51:51.367748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:51:51.367764 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:51:51.367920 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.367927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.369429 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.369432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.370524 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:51:51.370537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.371367 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.371370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:51:51.371461 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.371483 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:51:51.371541 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 17:51:51.371589 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":97,"threads_started":1}
I20260929 17:51:51.371763 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.366950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.183.194:34537 (local address 127.3.183.195:42951)
0929 17:51:51.367052 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.367055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.367429 (+ 374us) server_negotiation.cc:407] Connection header received
0929 17:51:51.367599 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.367603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.367649 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.367717 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:51:51.368056 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.368696 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.370666 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.371159 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.371236 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.371269 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.371583 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.371660 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:51:51.372054 24040 raft_consensus.cc:2781] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:51:51.372156 24040 raft_consensus.cc:482] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:51:51.372270 24040 raft_consensus.cc:3037] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:51.372478 23970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fa84a06c664478088d89d868d04badf" candidate_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" is_pre_election: true
I20260929 17:51:51.372749 23970 raft_consensus.cc:2454] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ef2c747e31413190e7fc4cfe83b7fc in term 0.
I20260929 17:51:51.373448 24040 raft_consensus.cc:504] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.373746 24040 leader_election.cc:288] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [CANDIDATE]: Term 1 election: Requested vote from peers a5b1cf4ba9f2440a841fa989f6dea233 (127.3.183.193:41773), 4e07d9a5aa66450f82ca9d5afeca7ff8 (127.3.183.195:42951)
I20260929 17:51:51.374051 23708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fa84a06c664478088d89d868d04badf" candidate_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b1cf4ba9f2440a841fa989f6dea233"
I20260929 17:51:51.374155 23708 raft_consensus.cc:3037] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:51.374145 23970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fa84a06c664478088d89d868d04badf" candidate_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8"
I20260929 17:51:51.374259 23970 raft_consensus.cc:3037] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:51.375131 23708 raft_consensus.cc:2454] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ef2c747e31413190e7fc4cfe83b7fc in term 1.
I20260929 17:51:51.375299 23970 raft_consensus.cc:2454] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ef2c747e31413190e7fc4cfe83b7fc in term 1.
I20260929 17:51:51.375448 23775 leader_election.cc:302] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b1cf4ba9f2440a841fa989f6dea233, c3ef2c747e31413190e7fc4cfe83b7fc; no voters:
I20260929 17:51:51.375615 24040 raft_consensus.cc:2781] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:51.375731 24040 raft_consensus.cc:686] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 1 LEADER]: Becoming Leader. State: Replica: c3ef2c747e31413190e7fc4cfe83b7fc, State: Running, Role: LEADER
I20260929 17:51:51.376085 24040 consensus_queue.cc:237] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } }
I20260929 17:51:51.379271 23565 catalog_manager.cc:5495] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc reported cstate change: term changed from 0 to 1, leader changed from <none> to c3ef2c747e31413190e7fc4cfe83b7fc (127.3.183.194). New cstate: current_term: 1 leader_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3ef2c747e31413190e7fc4cfe83b7fc" member_type: VOTER last_known_addr { host: "127.3.183.194" port: 37905 } health_report { overall_health: HEALTHY } } }
I20260929 17:51:51.390784 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.387209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.3.183.194:37905)
0929 17:51:51.387318 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:51.387322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:51.387336 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:51:51.387584 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:51.387586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:51.387643 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:51.387732 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:51:51.388024 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.388550 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.389417 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:51.389683 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:51.389759 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:51.390001 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:51:51.390433 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:51:51.390557 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:51.390607 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 17:51:51.390655 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 17:51:51.396497 23708 raft_consensus.cc:1261] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 1 FOLLOWER]: Refusing update from remote peer c3ef2c747e31413190e7fc4cfe83b7fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:51.396495 23970 raft_consensus.cc:1261] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 1 FOLLOWER]: Refusing update from remote peer c3ef2c747e31413190e7fc4cfe83b7fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:51:51.397140 24040 consensus_queue.cc:1035] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e07d9a5aa66450f82ca9d5afeca7ff8" member_type: VOTER last_known_addr { host: "127.3.183.195" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:51.397305 24039 consensus_queue.cc:1035] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b1cf4ba9f2440a841fa989f6dea233" member_type: VOTER last_known_addr { host: "127.3.183.193" port: 41773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:51:51.405143 24046 mvcc.cc:205] Tried to move back new op lower bound from 7334724859473330176 to 7334724859399417856. Current Snapshot: MvccSnapshot[applied={T|T < 7334724859473330176}]
I20260929 17:51:51.407023 24048 mvcc.cc:205] Tried to move back new op lower bound from 7334724859473330176 to 7334724859399417856. Current Snapshot: MvccSnapshot[applied={T|T < 7334724859473330176}]
I20260929 17:51:51.407153 24047 mvcc.cc:205] Tried to move back new op lower bound from 7334724859473330176 to 7334724859399417856. Current Snapshot: MvccSnapshot[applied={T|T < 7334724859473330176}]
I20260929 17:51:51.415655 23564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:51.415840 23564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:51:51.418202 23564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 183769e88b2647049eaeda0936c9499a: Sending DeleteTablet for 3 replicas of tablet 4fa84a06c664478088d89d868d04badf
I20260929 17:51:51.419054 23688 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fa84a06c664478088d89d868d04badf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:51 UTC) from {username='slave'} at 127.0.0.1:55024
I20260929 17:51:51.419052 23819 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fa84a06c664478088d89d868d04badf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:51 UTC) from {username='slave'} at 127.0.0.1:34640
I20260929 17:51:51.419538 24055 tablet_replica.cc:330] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233: stopping tablet replica
I20260929 17:51:51.419867 23950 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fa84a06c664478088d89d868d04badf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:51:51 UTC) from {username='slave'} at 127.0.0.1:52174
I20260929 17:51:51.419957 24055 raft_consensus.cc:2229] T 4fa84a06c664478088d89d868d04badf P a5b1cf4ba9f2440a841fa989f6dea233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:51.419986 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 23626
I20260929 17:51:51.423976 24056 tablet_replica.cc:330] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: stopping tablet replica
I20260929 17:51:51.424271 24057 tablet_replica.cc:330] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: stopping tablet replica
I20260929 17:51:51.424291 24056 raft_consensus.cc:2229] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:51:51.424640 24057 raft_consensus.cc:2229] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:51:51.424710 24056 raft_consensus.cc:2258] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:51.425007 24057 raft_consensus.cc:2258] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:51:51.426574 24056 ts_tablet_manager.cc:1905] T 4fa84a06c664478088d89d868d04badf P c3ef2c747e31413190e7fc4cfe83b7fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:51:51.426600 24057 ts_tablet_manager.cc:1905] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:51:51.427469 23553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.183.193:41773 (error 108)
I20260929 17:51:51.428330 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 23757
I20260929 17:51:51.429924 24057 ts_tablet_manager.cc:1918] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:51:51.430032 24057 log.cc:1196] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Deleting WAL directory at /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704058755242-3807-0/minicluster-data/ts-2/wal/wals/4fa84a06c664478088d89d868d04badf
I20260929 17:51:51.430296 24057 ts_tablet_manager.cc:1939] T 4fa84a06c664478088d89d868d04badf P 4e07d9a5aa66450f82ca9d5afeca7ff8: Deleting consensus metadata
I20260929 17:51:51.430958 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.430658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.193:41773 (local address 127.0.0.1:55032)
0929 17:51:51.430795 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.430870 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.193:41773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 17:51:51.431200 23553 catalog_manager.cc:4584] TS a5b1cf4ba9f2440a841fa989f6dea233 (127.3.183.193:41773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4fa84a06c664478088d89d868d04badf: Network error: Client connection negotiation failed: client connection to 127.3.183.193:41773: connect: Connection refused (error 111)
I20260929 17:51:51.431497 23553 catalog_manager.cc:4841] TS 4e07d9a5aa66450f82ca9d5afeca7ff8 (127.3.183.195:42951): tablet 4fa84a06c664478088d89d868d04badf (table test-table [id=f1b61f1e16914f73974c46f90b765cf9]) successfully deleted
I20260929 17:51:51.434072 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 23888
I20260929 17:51:51.437278 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:51.434153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.194:37905 (local address 127.0.0.1:34656)
0929 17:51:51.437142 (+ 2989us) negotiation.cc:107] Waiting for socket to connect
0929 17:51:51.437199 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.194:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2895}
I20260929 17:51:51.440445 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 23535
2026-09-29T17:51:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (969 ms)
[----------] 6 tests from EncryptionPolicyTest (10064 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:51 dist-test-slave-wbpt krb5kdc[24061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:51 dist-test-slave-wbpt krb5kdc[24061](info): set up 2 sockets
Sep 29 17:51:51 dist-test-slave-wbpt krb5kdc[24061](info): commencing operation
krb5kdc: starting...
W20260929 17:51:52.403635 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:52 dist-test-slave-wbpt krb5kdc[24061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:51:52.520826 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:52.628562 24074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:52.628829 24074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:52.628880 24074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:52.633008 24074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:52.633083 24074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:52.633101 24074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:52.633117 24074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:52.633131 24074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:52.638981 24074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:52.640179 24074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:52.641261 24074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:52.649788 24082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:52.649783 24079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:52.649979 24074 server_base.cc:1034] running on GCE node
W20260929 17:51:52.649788 24080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:52.650666 24074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:52.651823 24074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:52.652983 24074 hybrid_clock.cc:648] HybridClock initialized: now 1790704312652957 us; error 38 us; skew 500 ppm
Sep 29 17:51:52 dist-test-slave-wbpt krb5kdc[24061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:52.655709 24074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:51:52.656692 24074 webserver.cc:460] Webserver started at http://127.0.0.1:45495/ using document root <none> and password file <none>
I20260929 17:51:52.657245 24074 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:52.657312 24074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:52.657510 24074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:52.659579 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "06228f98b26a40928195e92037029719"
format_stamp: "Formatted at 2026-09-29 17:51:52 on dist-test-slave-wbpt"
server_key: "45d31caab8204bfb9ab045bdad370d73"
server_key_iv: "2aaaa0edaafeab04e72289e50da889dd"
server_key_version: "encryptionkey@0"
I20260929 17:51:52.660175 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "06228f98b26a40928195e92037029719"
format_stamp: "Formatted at 2026-09-29 17:51:52 on dist-test-slave-wbpt"
server_key: "45d31caab8204bfb9ab045bdad370d73"
server_key_iv: "2aaaa0edaafeab04e72289e50da889dd"
server_key_version: "encryptionkey@0"
I20260929 17:51:52.663905 24074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:51:52.666280 24089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:52.667490 24074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:52.667621 24074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "06228f98b26a40928195e92037029719"
format_stamp: "Formatted at 2026-09-29 17:51:52 on dist-test-slave-wbpt"
server_key: "45d31caab8204bfb9ab045bdad370d73"
server_key_iv: "2aaaa0edaafeab04e72289e50da889dd"
server_key_version: "encryptionkey@0"
I20260929 17:51:52.667719 24074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:52.684095 24074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:52.687501 24074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:52.687696 24074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:52.694759 24074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40025
I20260929 17:51:52.694787 24141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40025 every 8 connection(s)
I20260929 17:51:52.695730 24074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:52.696918 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24074
I20260929 17:51:52.697057 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:52.697335 3807 external_mini_cluster.cc:1428] Setting key 6ff93680920a61d1b09a6f97871d2759
I20260929 17:51:52.698781 24142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:52.705407 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719: Bootstrap starting.
Sep 29 17:51:52 dist-test-slave-wbpt krb5kdc[24061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:51:52.707887 24142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:52.708819 24142 log.cc:824] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:52.711129 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719: No bootstrap required, opened a new log
I20260929 17:51:52.712862 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:52.698955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57481 (local address 127.0.0.1:40025)
0929 17:51:52.699361 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:52.699380 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:52.699420 (+ 40us) server_negotiation.cc:407] Connection header received
0929 17:51:52.700314 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:52.700338 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:52.700766 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:52.701143 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:52.701811 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:52.703269 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:52.704220 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:52.704889 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:52.705052 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:52.707629 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:52.707635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:52.707652 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:52.707682 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:52.709466 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:52.710103 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:52.710109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:52.710114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:52.710249 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:52.710618 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:52.710621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:52.710623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:52.711132 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:52.711333 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:52.711652 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 17:51:52.711917 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260929 17:51:52.715490 24142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06228f98b26a40928195e92037029719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260929 17:51:52.715718 24142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:52.715797 24142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06228f98b26a40928195e92037029719, State: Initialized, Role: FOLLOWER
I20260929 17:51:52.715965 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:51:52.716291 24142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06228f98b26a40928195e92037029719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260929 17:51:52.716465 24142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:52.716528 24142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:52.716619 24142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:52.717908 24142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06228f98b26a40928195e92037029719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260929 17:51:52.718261 24142 leader_election.cc:302] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06228f98b26a40928195e92037029719; no voters:
I20260929 17:51:52.718580 24142 leader_election.cc:288] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:52.718678 24152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:52.718909 24152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [term 1 LEADER]: Becoming Leader. State: Replica: 06228f98b26a40928195e92037029719, State: Running, Role: LEADER
I20260929 17:51:52.719262 24152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06228f98b26a40928195e92037029719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } }
I20260929 17:51:52.720028 24142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:52.720778 24153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06228f98b26a40928195e92037029719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06228f98b26a40928195e92037029719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } } }
I20260929 17:51:52.720896 24153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:52.721202 24154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06228f98b26a40928195e92037029719. Latest consensus state: current_term: 1 leader_uuid: "06228f98b26a40928195e92037029719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06228f98b26a40928195e92037029719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40025 } } }
I20260929 17:51:52.721287 24154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719 [sys.catalog]: This master's current role is: LEADER
W20260929 17:51:52.725178 24168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:51:52.725262 24168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:51:52.725347 24161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:52.726163 24161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:52.726830 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:52.718119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55155 (local address 127.0.0.1:40025)
0929 17:51:52.718240 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:52.718244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:52.718529 (+ 285us) server_negotiation.cc:407] Connection header received
0929 17:51:52.718643 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:52.718645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:52.718693 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:52.718761 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:52.719604 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:52.720362 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:52.721478 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:52.721802 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:52.721907 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:52.723625 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:52.723628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:52.723630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:52.723652 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:52.725074 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:52.725584 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:52.725588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:52.725590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:52.725676 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:52.725986 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:52.725989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:52.725991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:52.726169 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:52.726297 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:52.726596 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 17:51:52.726685 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:51:52.732167 24161 catalog_manager.cc:1347] Generated new cluster ID: e9eb1785a65e4debab15f028fdd2c930
I20260929 17:51:52.732291 24161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:52.751107 24161 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:52.752466 24161 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:52.773890 24161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06228f98b26a40928195e92037029719: Generated new TSK 0
I20260929 17:51:52.774792 24161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:52.781569 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24074
W20260929 17:51:52.786765 24059 connection.cc:535] client connection to 127.0.0.1:40025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40025): Connection reset by peer (error 104)
2026-09-29T17:51:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:52 dist-test-slave-wbpt krb5kdc[24173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:52 dist-test-slave-wbpt krb5kdc[24173](info): set up 2 sockets
Sep 29 17:51:52 dist-test-slave-wbpt krb5kdc[24173](info): commencing operation
krb5kdc: starting...
W20260929 17:51:53.743745 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:53 dist-test-slave-wbpt krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:51:53.863494 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:53.971581 24186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:53.971887 24186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:53.971930 24186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:53.975936 24186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:53.976043 24186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:53.976081 24186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:53.976097 24186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:53.976118 24186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:53.981536 24186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:53.982594 24186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:53.983578 24186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:53.992162 24194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:53.992290 24191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:53.992290 24192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:53.992538 24186 server_base.cc:1034] running on GCE node
I20260929 17:51:53.993119 24186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:53.994242 24186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:53.995412 24186 hybrid_clock.cc:648] HybridClock initialized: now 1790704313995391 us; error 38 us; skew 500 ppm
Sep 29 17:51:53 dist-test-slave-wbpt krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:53.998077 24186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:51:53.998934 24186 webserver.cc:460] Webserver started at http://127.0.0.1:33275/ using document root <none> and password file <none>
I20260929 17:51:53.999461 24186 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:53.999529 24186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:53.999717 24186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:54.001838 24186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "60cc2976d46a4b1dace1d6632de1a95e"
format_stamp: "Formatted at 2026-09-29 17:51:53 on dist-test-slave-wbpt"
server_key: "5024530efaa07ac05ac59cf4123f6f48"
server_key_iv: "cdfe39a5e6092cd0f27b446f134d2327"
server_key_version: "encryptionkey@0"
I20260929 17:51:54.002390 24186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "60cc2976d46a4b1dace1d6632de1a95e"
format_stamp: "Formatted at 2026-09-29 17:51:53 on dist-test-slave-wbpt"
server_key: "5024530efaa07ac05ac59cf4123f6f48"
server_key_iv: "cdfe39a5e6092cd0f27b446f134d2327"
server_key_version: "encryptionkey@0"
I20260929 17:51:54.006114 24186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:51:54.008499 24201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:54.009809 24186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:51:54.009949 24186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "60cc2976d46a4b1dace1d6632de1a95e"
format_stamp: "Formatted at 2026-09-29 17:51:53 on dist-test-slave-wbpt"
server_key: "5024530efaa07ac05ac59cf4123f6f48"
server_key_iv: "cdfe39a5e6092cd0f27b446f134d2327"
server_key_version: "encryptionkey@0"
I20260929 17:51:54.010051 24186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:54.035137 24186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:54.038374 24186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:54.038566 24186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:54.045887 24186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40939
I20260929 17:51:54.045941 24253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40939 every 8 connection(s)
I20260929 17:51:54.046875 24186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:54.049721 24254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:54.049960 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24186
I20260929 17:51:54.050096 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:54.050565 3807 external_mini_cluster.cc:1428] Setting key 7a0e7924d08a50ea70efb6de38154562
I20260929 17:51:54.055866 24254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e: Bootstrap starting.
I20260929 17:51:54.058329 24254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:54.059072 24254 log.cc:824] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:54.060019 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:54.052176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53141 (local address 127.0.0.1:40939)
0929 17:51:54.052671 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:54.052686 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:54.052722 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:51:54.053379 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:54.053396 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:54.053768 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:54.054096 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:54.054512 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:54.055500 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:54.056493 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:54.056986 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:54.057149 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:54.058137 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:54.058142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:54.058156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:54.058602 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:54.058666 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:54.058690 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:54.058834 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:54.059067 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 17:51:54.059219 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":188,"threads_started":1}
I20260929 17:51:54.061293 24254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e: No bootstrap required, opened a new log
I20260929 17:51:54.063053 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:51:54.065426 24254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc2976d46a4b1dace1d6632de1a95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260929 17:51:54.065670 24254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:54.065726 24254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cc2976d46a4b1dace1d6632de1a95e, State: Initialized, Role: FOLLOWER
I20260929 17:51:54.066148 24254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc2976d46a4b1dace1d6632de1a95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260929 17:51:54.066278 24254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:54.066350 24254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:54.066447 24254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:54.067360 24254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc2976d46a4b1dace1d6632de1a95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260929 17:51:54.067682 24254 leader_election.cc:302] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60cc2976d46a4b1dace1d6632de1a95e; no voters:
I20260929 17:51:54.068019 24254 leader_election.cc:288] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:54.068140 24264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:54.068372 24264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [term 1 LEADER]: Becoming Leader. State: Replica: 60cc2976d46a4b1dace1d6632de1a95e, State: Running, Role: LEADER
I20260929 17:51:54.068743 24264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc2976d46a4b1dace1d6632de1a95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260929 17:51:54.069145 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:54.065071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43873 (local address 127.0.0.1:40939)
0929 17:51:54.065186 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:54.065191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:54.065392 (+ 201us) server_negotiation.cc:407] Connection header received
0929 17:51:54.065510 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:54.065513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:54.065556 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:54.065616 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:54.065880 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:54.066456 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:54.067233 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:54.067491 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:54.067576 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:54.068579 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:54.068583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:54.068585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:54.068598 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:54.068626 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:54.068638 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:54.068724 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:54.068983 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 17:51:54.069022 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:51:54.069350 24254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:54.069943 24264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60cc2976d46a4b1dace1d6632de1a95e. Latest consensus state: current_term: 1 leader_uuid: "60cc2976d46a4b1dace1d6632de1a95e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc2976d46a4b1dace1d6632de1a95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } } }
I20260929 17:51:54.070039 24264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:54.070627 24265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60cc2976d46a4b1dace1d6632de1a95e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60cc2976d46a4b1dace1d6632de1a95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } } }
I20260929 17:51:54.070719 24265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:54.070972 24270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:54.073798 24270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:54.079420 24270 catalog_manager.cc:1347] Generated new cluster ID: 1beeaecf1d8e4e81a0226c2963ce5123
I20260929 17:51:54.079493 24270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:54.091697 24270 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:54.092792 24270 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:54.103112 24270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60cc2976d46a4b1dace1d6632de1a95e: Generated new TSK 0
I20260929 17:51:54.103906 24270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:54.117432 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24186
W20260929 17:51:54.122527 24171 connection.cc:535] client connection to 127.0.0.1:40939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40939): Connection reset by peer (error 104)
2026-09-29T17:51:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:54 dist-test-slave-wbpt krb5kdc[24285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:54 dist-test-slave-wbpt krb5kdc[24285](info): set up 2 sockets
Sep 29 17:51:54 dist-test-slave-wbpt krb5kdc[24285](info): commencing operation
krb5kdc: starting...
W20260929 17:51:55.079830 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:55 dist-test-slave-wbpt krb5kdc[24285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:51:55.196565 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:55.305073 24298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:55.305354 24298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:55.305413 24298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:55.309648 24298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:55.309717 24298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:55.309743 24298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:55.309773 24298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:55.309803 24298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:55.315147 24298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:55.316233 24298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:55.317248 24298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:55.325454 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:55.325551 24298 server_base.cc:1034] running on GCE node
W20260929 17:51:55.325500 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:55.325492 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:55.326210 24298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:55.327286 24298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:55.328433 24298 hybrid_clock.cc:648] HybridClock initialized: now 1790704315328409 us; error 36 us; skew 500 ppm
Sep 29 17:51:55 dist-test-slave-wbpt krb5kdc[24285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:55.331007 24298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:51:55.331902 24298 webserver.cc:460] Webserver started at http://127.0.0.1:32943/ using document root <none> and password file <none>
I20260929 17:51:55.332394 24298 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:55.332445 24298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:55.332610 24298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:55.334560 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "1ec4a0d68c064300b2c7e6062f14666e"
format_stamp: "Formatted at 2026-09-29 17:51:55 on dist-test-slave-wbpt"
server_key: "92ec0ced6d7e19e2c1a48fe65c474825"
server_key_iv: "e83bac7bc08f0132832d99cec55a613f"
server_key_version: "encryptionkey@0"
I20260929 17:51:55.335048 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "1ec4a0d68c064300b2c7e6062f14666e"
format_stamp: "Formatted at 2026-09-29 17:51:55 on dist-test-slave-wbpt"
server_key: "92ec0ced6d7e19e2c1a48fe65c474825"
server_key_iv: "e83bac7bc08f0132832d99cec55a613f"
server_key_version: "encryptionkey@0"
I20260929 17:51:55.338701 24298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 17:51:55.340987 24313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:55.342125 24298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:51:55.342224 24298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "1ec4a0d68c064300b2c7e6062f14666e"
format_stamp: "Formatted at 2026-09-29 17:51:55 on dist-test-slave-wbpt"
server_key: "92ec0ced6d7e19e2c1a48fe65c474825"
server_key_iv: "e83bac7bc08f0132832d99cec55a613f"
server_key_version: "encryptionkey@0"
I20260929 17:51:55.342320 24298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:55.346942 24298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:55.349766 24298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:55.349917 24298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:55.356475 24298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41161
I20260929 17:51:55.356498 24365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41161 every 8 connection(s)
I20260929 17:51:55.357378 24298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:55.360080 24366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:55.362156 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24298
I20260929 17:51:55.362274 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:55.362529 3807 external_mini_cluster.cc:1428] Setting key b8c626c7475433c8eb8ea5cc766d620f
I20260929 17:51:55.365664 24366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e: Bootstrap starting.
I20260929 17:51:55.367482 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:55.363899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47599 (local address 127.0.0.1:41161)
0929 17:51:55.364345 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:55.364359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:55.364394 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:51:55.365034 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:55.365050 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:55.365335 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:55.365641 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:55.366362 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:55.366367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:55.366380 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:55.366390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:55.366442 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:55.366461 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:55.366580 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:55.366663 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 17:51:55.366811 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260929 17:51:55.367964 24366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:55.368651 24366 log.cc:824] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:55.370103 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:51:55.370472 24366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e: No bootstrap required, opened a new log
I20260929 17:51:55.373528 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:55.371980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60591 (local address 127.0.0.1:41161)
0929 17:51:55.372079 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:55.372082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:55.372298 (+ 216us) server_negotiation.cc:407] Connection header received
0929 17:51:55.372408 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:55.372411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:55.372442 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:55.372543 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:55.373177 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:55.373180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:55.373183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:55.373194 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:55.373220 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:55.373231 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:55.373298 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:55.373370 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 17:51:55.373407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 17:51:55.374234 24366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4a0d68c064300b2c7e6062f14666e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260929 17:51:55.374487 24366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:55.374553 24366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec4a0d68c064300b2c7e6062f14666e, State: Initialized, Role: FOLLOWER
I20260929 17:51:55.374979 24366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4a0d68c064300b2c7e6062f14666e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260929 17:51:55.375105 24366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:55.375165 24366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:55.375260 24366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:55.376219 24366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4a0d68c064300b2c7e6062f14666e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260929 17:51:55.376546 24366 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec4a0d68c064300b2c7e6062f14666e; no voters:
I20260929 17:51:55.376821 24366 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:55.376921 24376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:55.377149 24376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [term 1 LEADER]: Becoming Leader. State: Replica: 1ec4a0d68c064300b2c7e6062f14666e, State: Running, Role: LEADER
I20260929 17:51:55.377460 24376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4a0d68c064300b2c7e6062f14666e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260929 17:51:55.377955 24366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:55.378937 24377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec4a0d68c064300b2c7e6062f14666e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4a0d68c064300b2c7e6062f14666e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } } }
I20260929 17:51:55.379076 24377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:55.379354 24378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec4a0d68c064300b2c7e6062f14666e. Latest consensus state: current_term: 1 leader_uuid: "1ec4a0d68c064300b2c7e6062f14666e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4a0d68c064300b2c7e6062f14666e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } } }
I20260929 17:51:55.379436 24378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:55.379468 24385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:55.382057 24385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:55.387415 24385 catalog_manager.cc:1347] Generated new cluster ID: 3ddd3b3f0f1c4c31806ac63c3a849e93
I20260929 17:51:55.387480 24385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:55.412730 24385 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:55.413102 24385 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:55.419351 24385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec4a0d68c064300b2c7e6062f14666e: Generated new TSK 0
I20260929 17:51:55.420116 24385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:55.423084 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24298
2026-09-29T17:51:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:55 dist-test-slave-wbpt krb5kdc[24397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:55 dist-test-slave-wbpt krb5kdc[24397](info): set up 2 sockets
Sep 29 17:51:55 dist-test-slave-wbpt krb5kdc[24397](info): commencing operation
krb5kdc: starting...
W20260929 17:51:56.375659 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:56 dist-test-slave-wbpt krb5kdc[24397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:51:56.492727 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:56.600440 24410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:56.600721 24410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:56.600764 24410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:56.604887 24410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:56.604964 24410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:56.604983 24410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:56.605000 24410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:56.605013 24410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:56.610497 24410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:56.611596 24410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:56.612641 24410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:56.620944 24416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:56.620939 24415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:56.621078 24410 server_base.cc:1034] running on GCE node
W20260929 17:51:56.621008 24418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:56.621791 24410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:56.622929 24410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:56.624087 24410 hybrid_clock.cc:648] HybridClock initialized: now 1790704316624072 us; error 33 us; skew 500 ppm
Sep 29 17:51:56 dist-test-slave-wbpt krb5kdc[24397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:56.626706 24410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:51:56.627527 24410 webserver.cc:460] Webserver started at http://127.0.0.1:42767/ using document root <none> and password file <none>
I20260929 17:51:56.628077 24410 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:56.628139 24410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:56.628297 24410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:56.630259 24410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "0b8094a0886c4a59a983776f7d67e593"
format_stamp: "Formatted at 2026-09-29 17:51:56 on dist-test-slave-wbpt"
server_key: "3eeb135757a60dade30359e63da1e293"
server_key_iv: "04d7676642be16efd12cab16bc29ab17"
server_key_version: "encryptionkey@0"
I20260929 17:51:56.630750 24410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "0b8094a0886c4a59a983776f7d67e593"
format_stamp: "Formatted at 2026-09-29 17:51:56 on dist-test-slave-wbpt"
server_key: "3eeb135757a60dade30359e63da1e293"
server_key_iv: "04d7676642be16efd12cab16bc29ab17"
server_key_version: "encryptionkey@0"
I20260929 17:51:56.634327 24410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:51:56.636746 24425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:56.638005 24410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:56.638105 24410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "0b8094a0886c4a59a983776f7d67e593"
format_stamp: "Formatted at 2026-09-29 17:51:56 on dist-test-slave-wbpt"
server_key: "3eeb135757a60dade30359e63da1e293"
server_key_iv: "04d7676642be16efd12cab16bc29ab17"
server_key_version: "encryptionkey@0"
I20260929 17:51:56.638195 24410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:56.654335 24410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:56.657303 24410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:56.657467 24410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:56.663985 24410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44271
I20260929 17:51:56.664011 24477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44271 every 8 connection(s)
I20260929 17:51:56.664963 24410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:56.667630 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:56.668502 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24410
I20260929 17:51:56.668637 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:56.668931 3807 external_mini_cluster.cc:1428] Setting key 14c1397d7d8c2787c92973cc178bc8b9
I20260929 17:51:56.673703 24478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593: Bootstrap starting.
I20260929 17:51:56.675933 24478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:56.676702 24478 log.cc:824] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:51:56 dist-test-slave-wbpt krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:51:56.678660 24478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593: No bootstrap required, opened a new log
I20260929 17:51:56.682617 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:56.670419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58297 (local address 127.0.0.1:44271)
0929 17:51:56.670875 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:56.670890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:56.670928 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:51:56.671592 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:56.671612 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:56.672006 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:56.672316 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:56.673119 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:56.674107 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:56.675031 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:56.675531 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:56.675680 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:56.678198 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:56.678205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:56.678222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:56.678252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:56.679902 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:56.680376 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:56.680380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:56.680384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:56.680521 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:56.680775 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:56.680777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:56.680778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:56.681145 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:56.681284 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:56.681587 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 17:51:56.681870 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260929 17:51:56.682497 24478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8094a0886c4a59a983776f7d67e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260929 17:51:56.682725 24478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:56.682776 24478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8094a0886c4a59a983776f7d67e593, State: Initialized, Role: FOLLOWER
I20260929 17:51:56.683187 24478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8094a0886c4a59a983776f7d67e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260929 17:51:56.683310 24478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:56.683372 24478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:56.683494 24478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:56.684401 24478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8094a0886c4a59a983776f7d67e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260929 17:51:56.684784 24478 leader_election.cc:302] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b8094a0886c4a59a983776f7d67e593; no voters:
I20260929 17:51:56.685079 24478 leader_election.cc:288] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:56.685173 24483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:56.685349 24483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [term 1 LEADER]: Becoming Leader. State: Replica: 0b8094a0886c4a59a983776f7d67e593, State: Running, Role: LEADER
I20260929 17:51:56.685540 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:51:56.685645 24483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8094a0886c4a59a983776f7d67e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260929 17:51:56.686335 24478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:56.686781 24483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b8094a0886c4a59a983776f7d67e593. Latest consensus state: current_term: 1 leader_uuid: "0b8094a0886c4a59a983776f7d67e593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8094a0886c4a59a983776f7d67e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } } }
I20260929 17:51:56.686882 24483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:56.687134 24484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b8094a0886c4a59a983776f7d67e593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8094a0886c4a59a983776f7d67e593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } } }
I20260929 17:51:56.687234 24484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593 [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:56.687253 24491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:56.690487 24491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:56.696179 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:56.687769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44097 (local address 127.0.0.1:44271)
0929 17:51:56.687882 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:56.687885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:56.687945 (+ 60us) server_negotiation.cc:407] Connection header received
0929 17:51:56.688138 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:56.688141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:56.688186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:56.688251 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:56.689050 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:56.689810 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:56.690924 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:56.691214 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:56.691273 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:56.692766 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:56.692770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:56.692773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:51:56.692800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:56.694386 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:56.694828 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:56.694831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:56.694833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:56.694923 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:51:56.695238 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:51:56.695241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:51:56.695243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:51:56.695469 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:56.695607 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:56.695906 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 17:51:56.696028 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:51:56.696553 24491 catalog_manager.cc:1347] Generated new cluster ID: efc1b1088091484da605b52951267318
I20260929 17:51:56.696621 24491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:56.708724 24491 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:56.709710 24491 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:56.714366 24491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b8094a0886c4a59a983776f7d67e593: Generated new TSK 0
I20260929 17:51:56.715131 24491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:56.719641 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24410
2026-09-29T17:51:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:56 dist-test-slave-wbpt krb5kdc[24509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:56 dist-test-slave-wbpt krb5kdc[24509](info): set up 2 sockets
Sep 29 17:51:56 dist-test-slave-wbpt krb5kdc[24509](info): commencing operation
krb5kdc: starting...
W20260929 17:51:57.683727 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:57 dist-test-slave-wbpt krb5kdc[24509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:51:57.808648 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:57.919602 24522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:57.919929 24522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:57.919972 24522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:57.924067 24522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:57.924132 24522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:57.924150 24522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:57.924166 24522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:57.924187 24522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:57.929541 24522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:57.930572 24522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:57.931527 24522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:51:57.939980 24522 server_base.cc:1034] running on GCE node
W20260929 17:51:57.939925 24527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:57.939920 24528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:57.939920 24530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:57.940860 24522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:57.941988 24522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:57.943171 24522 hybrid_clock.cc:648] HybridClock initialized: now 1790704317943154 us; error 29 us; skew 500 ppm
Sep 29 17:51:57 dist-test-slave-wbpt krb5kdc[24509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:57.945971 24522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:51:57.946887 24522 webserver.cc:460] Webserver started at http://127.0.0.1:45277/ using document root <none> and password file <none>
I20260929 17:51:57.947399 24522 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:57.947459 24522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:57.947624 24522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:57.949716 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "8a59a6ec7b5d4dad9f65a000f3db711d"
format_stamp: "Formatted at 2026-09-29 17:51:57 on dist-test-slave-wbpt"
server_key: "4337174217d2e8252028c3171fedb03e"
server_key_iv: "e78e455708446e844e4fa24b13041e58"
server_key_version: "encryptionkey@0"
I20260929 17:51:57.950253 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "8a59a6ec7b5d4dad9f65a000f3db711d"
format_stamp: "Formatted at 2026-09-29 17:51:57 on dist-test-slave-wbpt"
server_key: "4337174217d2e8252028c3171fedb03e"
server_key_iv: "e78e455708446e844e4fa24b13041e58"
server_key_version: "encryptionkey@0"
I20260929 17:51:57.954106 24522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 17:51:57.956664 24537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:57.957986 24522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:51:57.958127 24522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "8a59a6ec7b5d4dad9f65a000f3db711d"
format_stamp: "Formatted at 2026-09-29 17:51:57 on dist-test-slave-wbpt"
server_key: "4337174217d2e8252028c3171fedb03e"
server_key_iv: "e78e455708446e844e4fa24b13041e58"
server_key_version: "encryptionkey@0"
I20260929 17:51:57.958240 24522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:57.976850 24522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:57.979984 24522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:57.980190 24522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:57.987161 24522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42557
I20260929 17:51:57.987187 24589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42557 every 8 connection(s)
I20260929 17:51:57.988152 24522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:57.991067 24590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:57.995172 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24522
I20260929 17:51:57.995296 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:57.995632 3807 external_mini_cluster.cc:1428] Setting key 691d3d683df8c20f0a02e93d35c79a14
I20260929 17:51:57.997078 24590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d: Bootstrap starting.
I20260929 17:51:57.999370 24590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:58.000146 24590 log.cc:824] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:58.002054 24590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d: No bootstrap required, opened a new log
I20260929 17:51:58.004695 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:57.997098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45179 (local address 127.0.0.1:42557)
0929 17:51:57.997577 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:57.997593 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:57.997630 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:51:57.998318 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:57.998335 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:57.998721 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:57.999051 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:57.999471 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:58.000550 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:58.001448 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:58.001931 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:58.002076 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:58.002926 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:58.002931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:58.002945 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:58.003359 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:58.003422 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:58.003447 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:58.003533 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:58.003677 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 17:51:58.003922 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":147,"threads_started":1}
I20260929 17:51:58.005901 24590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260929 17:51:58.006181 24590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:58.006237 24590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a59a6ec7b5d4dad9f65a000f3db711d, State: Initialized, Role: FOLLOWER
I20260929 17:51:58.006685 24590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260929 17:51:58.006826 24590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:58.006892 24590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:58.006980 24590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:58.007620 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:51:58.008117 24590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260929 17:51:58.008447 24590 leader_election.cc:302] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a59a6ec7b5d4dad9f65a000f3db711d; no voters:
I20260929 17:51:58.008754 24590 leader_election.cc:288] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:58.009109 24600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:58.009290 24600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [term 1 LEADER]: Becoming Leader. State: Replica: 8a59a6ec7b5d4dad9f65a000f3db711d, State: Running, Role: LEADER
I20260929 17:51:58.009588 24600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260929 17:51:58.010125 24590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:58.010789 24600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a59a6ec7b5d4dad9f65a000f3db711d. Latest consensus state: current_term: 1 leader_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } } }
I20260929 17:51:58.010987 24600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:58.011174 24597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a59a6ec7b5d4dad9f65a000f3db711d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } } }
I20260929 17:51:58.011361 24597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:58.011405 24609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:58.014607 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:58.009961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42801 (local address 127.0.0.1:42557)
0929 17:51:58.010063 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:58.010068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:58.010114 (+ 46us) server_negotiation.cc:407] Connection header received
0929 17:51:58.010282 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:58.010286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:58.010333 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:58.010396 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:58.010676 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:58.011459 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:58.012606 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:51:58.012870 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:51:58.012973 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:51:58.013815 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:58.013819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:58.013822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:58.013836 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:58.013865 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:58.013880 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:58.013948 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:58.014413 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 17:51:58.014473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:51:58.014914 24609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:58.020754 24609 catalog_manager.cc:1347] Generated new cluster ID: ab5994be9a954b27b7310327e9d6af51
I20260929 17:51:58.020838 24609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:58.037329 24609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:58.038371 24609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:58.045732 24609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a59a6ec7b5d4dad9f65a000f3db711d: Generated new TSK 0
I20260929 17:51:58.046610 24609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:58.060237 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24522
2026-09-29T17:51:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:58 dist-test-slave-wbpt krb5kdc[24621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:58 dist-test-slave-wbpt krb5kdc[24621](info): set up 2 sockets
Sep 29 17:51:58 dist-test-slave-wbpt krb5kdc[24621](info): commencing operation
krb5kdc: starting...
W20260929 17:51:59.019723 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:51:59 dist-test-slave-wbpt krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:51:59.143481 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:51:59.272909 24634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:51:59.273212 24634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:51:59.273257 24634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:51:59.277364 24634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:51:59.277438 24634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:51:59.277457 24634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:51:59.277472 24634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:51:59.277486 24634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:51:59.283005 24634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:51:59.284173 24634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:51:59.285197 24634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:51:59.294198 24639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:59.294350 24634 server_base.cc:1034] running on GCE node
W20260929 17:51:59.294189 24642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:51:59.294194 24640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:51:59.295168 24634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:51:59.296492 24634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:51:59.297719 24634 hybrid_clock.cc:648] HybridClock initialized: now 1790704319297691 us; error 40 us; skew 500 ppm
Sep 29 17:51:59 dist-test-slave-wbpt krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:51:59.301047 24634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:51:59.302187 24634 webserver.cc:460] Webserver started at http://127.0.0.1:37473/ using document root <none> and password file <none>
I20260929 17:51:59.302801 24634 fs_manager.cc:359] Metadata directory not provided
I20260929 17:51:59.302870 24634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:51:59.303089 24634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:51:59.305387 24634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "50103f24ddb940ac9d085165afe6a39c"
format_stamp: "Formatted at 2026-09-29 17:51:59 on dist-test-slave-wbpt"
server_key: "2f291552532791a7a57459408f5e0232"
server_key_iv: "cd3c2df4784ab50601443b083fdcfc86"
server_key_version: "encryptionkey@0"
I20260929 17:51:59.305948 24634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "50103f24ddb940ac9d085165afe6a39c"
format_stamp: "Formatted at 2026-09-29 17:51:59 on dist-test-slave-wbpt"
server_key: "2f291552532791a7a57459408f5e0232"
server_key_iv: "cd3c2df4784ab50601443b083fdcfc86"
server_key_version: "encryptionkey@0"
I20260929 17:51:59.309805 24634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:51:59.312510 24649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:51:59.313823 24634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:51:59.313944 24634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "50103f24ddb940ac9d085165afe6a39c"
format_stamp: "Formatted at 2026-09-29 17:51:59 on dist-test-slave-wbpt"
server_key: "2f291552532791a7a57459408f5e0232"
server_key_iv: "cd3c2df4784ab50601443b083fdcfc86"
server_key_version: "encryptionkey@0"
I20260929 17:51:59.314062 24634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:51:59.318836 24634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:51:59.322085 24634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:51:59.322261 24634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:51:59.329517 24634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46865
I20260929 17:51:59.329588 24701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46865 every 8 connection(s)
I20260929 17:51:59.330521 24634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:51:59.333496 24702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:51:59.339679 24702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c: Bootstrap starting.
I20260929 17:51:59.340411 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24634
I20260929 17:51:59.340576 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:51:59.340903 3807 external_mini_cluster.cc:1428] Setting key 05033f78790dbb8d8f5e736aa5742818
I20260929 17:51:59.342976 24702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c: Neither blocks nor log segments found. Creating new log.
I20260929 17:51:59.343755 24702 log.cc:824] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:51:59.346040 24702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c: No bootstrap required, opened a new log
I20260929 17:51:59.346436 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:59.342381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49583 (local address 127.0.0.1:46865)
0929 17:51:59.342874 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:59.342890 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:59.342927 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:51:59.343676 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:59.343695 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:59.344061 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:59.344387 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:59.345191 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:59.345195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:59.345208 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:59.345218 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:59.345273 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:59.345292 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:59.345391 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:59.345522 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 17:51:59.345680 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260929 17:51:59.349529 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:51:59.350128 24702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50103f24ddb940ac9d085165afe6a39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46865 } }
I20260929 17:51:59.350394 24702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:51:59.350449 24702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50103f24ddb940ac9d085165afe6a39c, State: Initialized, Role: FOLLOWER
I20260929 17:51:59.350962 24702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50103f24ddb940ac9d085165afe6a39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46865 } }
I20260929 17:51:59.351118 24702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:51:59.351190 24702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:51:59.351292 24702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:51:59.352514 24702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50103f24ddb940ac9d085165afe6a39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46865 } }
I20260929 17:51:59.352941 24702 leader_election.cc:302] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50103f24ddb940ac9d085165afe6a39c; no voters:
I20260929 17:51:59.353305 24702 leader_election.cc:288] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:51:59.353394 24712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:51:59.353513 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:51:59.351680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49073 (local address 127.0.0.1:46865)
0929 17:51:59.351817 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:51:59.351822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:51:59.352052 (+ 230us) server_negotiation.cc:407] Connection header received
0929 17:51:59.352179 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:51:59.352182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:51:59.352221 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:51:59.352287 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:51:59.353017 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:51:59.353021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:51:59.353023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:51:59.353036 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:51:59.353065 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:51:59.353075 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:51:59.353205 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 17:51:59.353284 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 17:51:59.353328 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:51:59.353844 24712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [term 1 LEADER]: Becoming Leader. State: Replica: 50103f24ddb940ac9d085165afe6a39c, State: Running, Role: LEADER
I20260929 17:51:59.354180 24712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50103f24ddb940ac9d085165afe6a39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46865 } }
I20260929 17:51:59.354718 24702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:51:59.355463 24712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50103f24ddb940ac9d085165afe6a39c. Latest consensus state: current_term: 1 leader_uuid: "50103f24ddb940ac9d085165afe6a39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50103f24ddb940ac9d085165afe6a39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46865 } } }
I20260929 17:51:59.355597 24712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [sys.catalog]: This master's current role is: LEADER
I20260929 17:51:59.355950 24713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50103f24ddb940ac9d085165afe6a39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50103f24ddb940ac9d085165afe6a39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46865 } } }
I20260929 17:51:59.356031 24713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c [sys.catalog]: This master's current role is: LEADER
W20260929 17:51:59.360056 24727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:51:59.360139 24727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:51:59.360272 24721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:51:59.361236 24721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:51:59.366922 24721 catalog_manager.cc:1347] Generated new cluster ID: d77535ca55734855b379a8d75760a5f4
I20260929 17:51:59.367013 24721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:51:59.395738 24721 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:51:59.396103 24721 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:51:59.407855 24721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50103f24ddb940ac9d085165afe6a39c: Generated new TSK 0
I20260929 17:51:59.408600 24721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:51:59.412855 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24634
2026-09-29T17:51:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:51:59 dist-test-slave-wbpt krb5kdc[24733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:51:59 dist-test-slave-wbpt krb5kdc[24733](info): set up 2 sockets
Sep 29 17:51:59 dist-test-slave-wbpt krb5kdc[24733](info): commencing operation
krb5kdc: starting...
W20260929 17:52:00.372040 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:00 dist-test-slave-wbpt krb5kdc[24733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:52:00.495733 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:00.608645 24746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:00.608958 24746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:00.609017 24746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:00.613293 24746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:00.613379 24746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:00.613407 24746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:00.613428 24746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:00.613448 24746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:00.619089 24746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:00.620366 24746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:00.621446 24746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:00.629843 24754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:00.629837 24752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:00.629977 24746 server_base.cc:1034] running on GCE node
W20260929 17:52:00.629866 24751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:00.630729 24746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:00.631938 24746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:00.633126 24746 hybrid_clock.cc:648] HybridClock initialized: now 1790704320633114 us; error 35 us; skew 500 ppm
Sep 29 17:52:00 dist-test-slave-wbpt krb5kdc[24733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:00.635865 24746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:52:00.636797 24746 webserver.cc:460] Webserver started at http://127.0.0.1:44521/ using document root <none> and password file <none>
I20260929 17:52:00.637365 24746 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:00.637434 24746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:00.637629 24746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:00.639721 24746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "ff4ea1b83b9740119935d84800406778"
format_stamp: "Formatted at 2026-09-29 17:52:00 on dist-test-slave-wbpt"
server_key: "95c88fe7fb152f7561fa37ad036d9625"
server_key_iv: "667bb8d493036f45716c9f3b59d48101"
server_key_version: "encryptionkey@0"
I20260929 17:52:00.640337 24746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "ff4ea1b83b9740119935d84800406778"
format_stamp: "Formatted at 2026-09-29 17:52:00 on dist-test-slave-wbpt"
server_key: "95c88fe7fb152f7561fa37ad036d9625"
server_key_iv: "667bb8d493036f45716c9f3b59d48101"
server_key_version: "encryptionkey@0"
I20260929 17:52:00.644201 24746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:52:00.646687 24761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:00.648070 24746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:00.648191 24746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "ff4ea1b83b9740119935d84800406778"
format_stamp: "Formatted at 2026-09-29 17:52:00 on dist-test-slave-wbpt"
server_key: "95c88fe7fb152f7561fa37ad036d9625"
server_key_iv: "667bb8d493036f45716c9f3b59d48101"
server_key_version: "encryptionkey@0"
I20260929 17:52:00.648300 24746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:00.663257 24746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:00.666335 24746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:00.666527 24746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:00.673238 24746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41457
I20260929 17:52:00.673255 24813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41457 every 8 connection(s)
I20260929 17:52:00.674201 24746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:00.677145 24814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:00.682292 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24746
I20260929 17:52:00.682430 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:00.682686 3807 external_mini_cluster.cc:1428] Setting key bfe2a5cdd13f055f4bd01d872947bc0f
I20260929 17:52:00.682816 24814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778: Bootstrap starting.
I20260929 17:52:00.685694 24814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:00.686585 24814 log.cc:824] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:00.688835 24814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778: No bootstrap required, opened a new log
Sep 29 17:52:00 dist-test-slave-wbpt krb5kdc[24733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:52:00.693128 24814 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4ea1b83b9740119935d84800406778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41457 } }
I20260929 17:52:00.693398 24814 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:00.693456 24814 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4ea1b83b9740119935d84800406778, State: Initialized, Role: FOLLOWER
I20260929 17:52:00.693984 24814 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4ea1b83b9740119935d84800406778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41457 } }
I20260929 17:52:00.694152 24814 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:00.694226 24814 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:00.694326 24814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:00.695470 24814 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4ea1b83b9740119935d84800406778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41457 } }
I20260929 17:52:00.695883 24814 leader_election.cc:302] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff4ea1b83b9740119935d84800406778; no voters:
I20260929 17:52:00.696206 24814 leader_election.cc:288] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:00.696309 24819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:00.696473 24819 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [term 1 LEADER]: Becoming Leader. State: Replica: ff4ea1b83b9740119935d84800406778, State: Running, Role: LEADER
I20260929 17:52:00.696847 24819 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4ea1b83b9740119935d84800406778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41457 } }
I20260929 17:52:00.697638 24814 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:00.698540 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:00.684022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:54187 (local address 127.0.0.1:41457)
0929 17:52:00.684438 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:00.684459 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:00.684502 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:52:00.685337 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:00.685362 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:00.685825 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:00.686213 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:00.686969 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:00.688323 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:00.689315 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:00.690002 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:00.690152 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:00.692872 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:00.692878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:00.692897 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:00.692931 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:00.695009 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:00.695553 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:00.695560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:00.695567 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:00.695711 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:00.696087 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:00.696091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:00.696093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:00.696668 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:00.696944 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:00.697321 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 17:52:00.697577 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260929 17:52:00.698844 24820 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff4ea1b83b9740119935d84800406778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4ea1b83b9740119935d84800406778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41457 } } }
I20260929 17:52:00.698976 24820 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:00.699295 24821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff4ea1b83b9740119935d84800406778. Latest consensus state: current_term: 1 leader_uuid: "ff4ea1b83b9740119935d84800406778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4ea1b83b9740119935d84800406778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41457 } } }
I20260929 17:52:00.699410 24821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:00.699681 24828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:00.702116 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:00.703142 24828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:00.712687 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:00.704305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:45425 (local address 127.0.0.1:41457)
0929 17:52:00.704417 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:00.704422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:00.705096 (+ 674us) server_negotiation.cc:407] Connection header received
0929 17:52:00.705204 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:00.705207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:00.705253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:00.705317 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:00.705985 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:00.706512 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:00.707454 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:00.707723 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:00.707893 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:00.709246 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:00.709251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:00.709254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:00.709282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:00.710819 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:00.711273 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:00.711277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:00.711279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:00.711372 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:00.711684 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:00.711687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:00.711689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:00.711955 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:00.712117 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:00.712413 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 17:52:00.712517 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 17:52:00.713625 24828 catalog_manager.cc:1347] Generated new cluster ID: 96dfc8f546ec436a8975d9d2989fe9e3
I20260929 17:52:00.713691 24828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:00.743682 24828 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:00.745110 24828 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:00.756475 24828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff4ea1b83b9740119935d84800406778: Generated new TSK 0
I20260929 17:52:00.757386 24828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:00.771560 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24746
W20260929 17:52:00.776852 24731 connection.cc:535] client connection to 127.0.0.1:41457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41457): Connection reset by peer (error 104)
2026-09-29T17:52:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:00 dist-test-slave-wbpt krb5kdc[24845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:00 dist-test-slave-wbpt krb5kdc[24845](info): set up 2 sockets
Sep 29 17:52:00 dist-test-slave-wbpt krb5kdc[24845](info): commencing operation
krb5kdc: starting...
W20260929 17:52:01.719825 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:01 dist-test-slave-wbpt krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:52:01.836639 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:01.946305 24858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:01.946611 24858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:01.946672 24858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:01.950726 24858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:01.950803 24858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:01.950830 24858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:01.950851 24858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:01.950872 24858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:01.956264 24858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:01.957388 24858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:01.958405 24858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:01.966774 24864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:01.966778 24866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:01.966885 24858 server_base.cc:1034] running on GCE node
W20260929 17:52:01.966782 24863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:01.967572 24858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:01.968699 24858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:01.969872 24858 hybrid_clock.cc:648] HybridClock initialized: now 1790704321969845 us; error 42 us; skew 500 ppm
Sep 29 17:52:01 dist-test-slave-wbpt krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:01.972743 24858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:52:01.973677 24858 webserver.cc:460] Webserver started at http://127.0.0.1:37385/ using document root <none> and password file <none>
I20260929 17:52:01.974223 24858 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:01.974290 24858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:01.974483 24858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:01.976632 24858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "b7e386ece46543e3857ca841d4d69284"
format_stamp: "Formatted at 2026-09-29 17:52:01 on dist-test-slave-wbpt"
server_key: "ea62563715cf3f44464f24832057910f"
server_key_iv: "baabe17bfa41e227bd7778b0242869df"
server_key_version: "encryptionkey@0"
I20260929 17:52:01.977208 24858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "b7e386ece46543e3857ca841d4d69284"
format_stamp: "Formatted at 2026-09-29 17:52:01 on dist-test-slave-wbpt"
server_key: "ea62563715cf3f44464f24832057910f"
server_key_iv: "baabe17bfa41e227bd7778b0242869df"
server_key_version: "encryptionkey@0"
I20260929 17:52:01.981003 24858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:52:01.983644 24873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:01.985144 24858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:52:01.985280 24858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "b7e386ece46543e3857ca841d4d69284"
format_stamp: "Formatted at 2026-09-29 17:52:01 on dist-test-slave-wbpt"
server_key: "ea62563715cf3f44464f24832057910f"
server_key_iv: "baabe17bfa41e227bd7778b0242869df"
server_key_version: "encryptionkey@0"
I20260929 17:52:01.985409 24858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:02.008127 24858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:02.012183 24858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:02.012399 24858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:02.019883 24858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41831
I20260929 17:52:02.019923 24925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41831 every 8 connection(s)
I20260929 17:52:02.020917 24858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:02.022665 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24858
I20260929 17:52:02.022809 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:02.023118 3807 external_mini_cluster.cc:1428] Setting key c0487c1d3fe5156e6c650ea90a7dbb25
I20260929 17:52:02.024238 24926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:02.029857 24926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284: Bootstrap starting.
I20260929 17:52:02.032044 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:02.024576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:52403 (local address 127.0.0.1:41831)
0929 17:52:02.025029 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:02.025044 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:02.025080 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:52:02.025716 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:02.025733 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:02.026072 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:02.026389 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:02.026804 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:02.027832 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:02.028735 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:02.029205 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:02.029353 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:02.030301 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:02.030306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:02.030356 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:02.030740 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:02.030797 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:02.030821 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:02.030932 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:02.031108 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 17:52:02.031241 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260929 17:52:02.032179 24926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:02.032889 24926 log.cc:824] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:02.035018 24926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284: No bootstrap required, opened a new log
I20260929 17:52:02.035102 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:02.039638 24926 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e386ece46543e3857ca841d4d69284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41831 } }
I20260929 17:52:02.039980 24926 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:02.040043 24926 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e386ece46543e3857ca841d4d69284, State: Initialized, Role: FOLLOWER
I20260929 17:52:02.040536 24926 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e386ece46543e3857ca841d4d69284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41831 } }
I20260929 17:52:02.040697 24926 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:02.040763 24926 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:02.040867 24926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:02.041515 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:02.037352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:37199 (local address 127.0.0.1:41831)
0929 17:52:02.037458 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:02.037461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:02.037696 (+ 235us) server_negotiation.cc:407] Connection header received
0929 17:52:02.037804 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:02.037807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:02.037853 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:02.037923 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:02.038238 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:02.038779 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:02.039873 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:02.040179 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:02.040269 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:02.041097 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:02.041100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:02.041102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:02.041114 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:02.041138 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:02.041148 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:02.041214 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:02.041349 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 17:52:02.041411 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:52:02.042042 24926 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e386ece46543e3857ca841d4d69284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41831 } }
I20260929 17:52:02.042403 24926 leader_election.cc:302] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e386ece46543e3857ca841d4d69284; no voters:
I20260929 17:52:02.042779 24926 leader_election.cc:288] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:02.042871 24936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:02.043105 24936 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [term 1 LEADER]: Becoming Leader. State: Replica: b7e386ece46543e3857ca841d4d69284, State: Running, Role: LEADER
I20260929 17:52:02.043494 24936 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e386ece46543e3857ca841d4d69284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41831 } }
I20260929 17:52:02.044049 24926 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:02.045220 24938 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e386ece46543e3857ca841d4d69284. Latest consensus state: current_term: 1 leader_uuid: "b7e386ece46543e3857ca841d4d69284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e386ece46543e3857ca841d4d69284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41831 } } }
I20260929 17:52:02.045236 24937 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7e386ece46543e3857ca841d4d69284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e386ece46543e3857ca841d4d69284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41831 } } }
I20260929 17:52:02.045452 24937 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:02.045435 24938 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:02.045857 24948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:02.049048 24948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:02.055274 24948 catalog_manager.cc:1347] Generated new cluster ID: 89514ce9a4094a8bb49f55593dee55fd
I20260929 17:52:02.055362 24948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:02.062311 24948 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:02.063601 24948 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:02.068653 24948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7e386ece46543e3857ca841d4d69284: Generated new TSK 0
I20260929 17:52:02.069551 24948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:02.074512 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24858
W20260929 17:52:02.079246 24843 connection.cc:535] client connection to 127.0.0.1:41831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41831): Connection reset by peer (error 104)
2026-09-29T17:52:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:02 dist-test-slave-wbpt krb5kdc[24957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:02 dist-test-slave-wbpt krb5kdc[24957](info): set up 2 sockets
Sep 29 17:52:02 dist-test-slave-wbpt krb5kdc[24957](info): commencing operation
krb5kdc: starting...
W20260929 17:52:03.027848 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:03 dist-test-slave-wbpt krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:52:03.151552 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:03.261546 24970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:03.261818 24970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:03.261881 24970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:03.265986 24970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:03.266063 24970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:03.266090 24970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:03.266110 24970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:03.266131 24970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:03.271624 24970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:03.272713 24970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:03.273733 24970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:03.282243 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:03.282305 24970 server_base.cc:1034] running on GCE node
W20260929 17:52:03.282258 24975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:03.282250 24976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:03.282976 24970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:03.284132 24970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:03.285283 24970 hybrid_clock.cc:648] HybridClock initialized: now 1790704323285256 us; error 40 us; skew 500 ppm
Sep 29 17:52:03 dist-test-slave-wbpt krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:03.287999 24970 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:52:03.288921 24970 webserver.cc:460] Webserver started at http://127.0.0.1:42955/ using document root <none> and password file <none>
I20260929 17:52:03.289460 24970 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:03.289526 24970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:03.289718 24970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:03.291744 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "ea99c1f53ac24540b03289d1518fb774"
format_stamp: "Formatted at 2026-09-29 17:52:03 on dist-test-slave-wbpt"
server_key: "e19fcbea86d54cc349988fba594cf2e5"
server_key_iv: "d2d7cff4b33681230f5c389604cb0346"
server_key_version: "encryptionkey@0"
I20260929 17:52:03.292285 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "ea99c1f53ac24540b03289d1518fb774"
format_stamp: "Formatted at 2026-09-29 17:52:03 on dist-test-slave-wbpt"
server_key: "e19fcbea86d54cc349988fba594cf2e5"
server_key_iv: "d2d7cff4b33681230f5c389604cb0346"
server_key_version: "encryptionkey@0"
I20260929 17:52:03.295933 24970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:52:03.298264 24985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:03.299450 24970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:52:03.299571 24970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "ea99c1f53ac24540b03289d1518fb774"
format_stamp: "Formatted at 2026-09-29 17:52:03 on dist-test-slave-wbpt"
server_key: "e19fcbea86d54cc349988fba594cf2e5"
server_key_iv: "d2d7cff4b33681230f5c389604cb0346"
server_key_version: "encryptionkey@0"
I20260929 17:52:03.299665 24970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:03.304116 24970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:03.306847 24970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:03.307014 24970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:03.313690 24970 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41709
I20260929 17:52:03.313737 25037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41709 every 8 connection(s)
I20260929 17:52:03.314594 24970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:03.317292 25038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:03.317909 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 24970
I20260929 17:52:03.318044 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:03.318358 3807 external_mini_cluster.cc:1428] Setting key cbb5e1c0acff66e963b2a5907366d8cf
I20260929 17:52:03.323289 25038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774: Bootstrap starting.
I20260929 17:52:03.323498 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:03.319738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:53069 (local address 127.0.0.1:41709)
0929 17:52:03.320268 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:03.320282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:03.320319 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:52:03.320986 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:03.321002 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:03.321279 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:03.321597 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:03.322332 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:03.322337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:03.322350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:03.322360 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:03.322412 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:03.322431 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:03.322561 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:03.322629 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 17:52:03.322784 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":128,"threads_started":1}
I20260929 17:52:03.325683 25038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:03.326198 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:03.326424 25038 log.cc:824] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:03.328413 25038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774: No bootstrap required, opened a new log
I20260929 17:52:03.329635 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:03.328072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:60943 (local address 127.0.0.1:41709)
0929 17:52:03.328169 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:03.328174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:03.328368 (+ 194us) server_negotiation.cc:407] Connection header received
0929 17:52:03.328542 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:03.328545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:03.328578 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:03.328638 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:03.329296 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:03.329299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:03.329302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:03.329313 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:03.329338 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:03.329348 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:03.329414 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:03.329485 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 17:52:03.329523 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 17:52:03.332168 25038 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea99c1f53ac24540b03289d1518fb774" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260929 17:52:03.332418 25038 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:03.332470 25038 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99c1f53ac24540b03289d1518fb774, State: Initialized, Role: FOLLOWER
I20260929 17:52:03.332885 25038 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea99c1f53ac24540b03289d1518fb774" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260929 17:52:03.333025 25038 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:03.333093 25038 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:03.333199 25038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:03.334164 25038 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea99c1f53ac24540b03289d1518fb774" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260929 17:52:03.334506 25038 leader_election.cc:302] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea99c1f53ac24540b03289d1518fb774; no voters:
I20260929 17:52:03.334810 25038 leader_election.cc:288] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:03.334928 25048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:03.335153 25048 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [term 1 LEADER]: Becoming Leader. State: Replica: ea99c1f53ac24540b03289d1518fb774, State: Running, Role: LEADER
I20260929 17:52:03.335520 25048 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea99c1f53ac24540b03289d1518fb774" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260929 17:52:03.336030 25038 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:03.337239 25049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea99c1f53ac24540b03289d1518fb774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea99c1f53ac24540b03289d1518fb774" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } } }
I20260929 17:52:03.337257 25050 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea99c1f53ac24540b03289d1518fb774. Latest consensus state: current_term: 1 leader_uuid: "ea99c1f53ac24540b03289d1518fb774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea99c1f53ac24540b03289d1518fb774" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } } }
I20260929 17:52:03.337383 25049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:03.337383 25050 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:03.337792 25057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:03.340821 25057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:03.346498 25057 catalog_manager.cc:1347] Generated new cluster ID: c744274c834d41f593ac42e78498582c
I20260929 17:52:03.346624 25057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:03.373126 25057 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:03.373503 25057 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:03.385854 25057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea99c1f53ac24540b03289d1518fb774: Generated new TSK 0
I20260929 17:52:03.386607 25057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:03.399670 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 24970
2026-09-29T17:52:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:03 dist-test-slave-wbpt krb5kdc[25069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:03 dist-test-slave-wbpt krb5kdc[25069](info): set up 2 sockets
Sep 29 17:52:03 dist-test-slave-wbpt krb5kdc[25069](info): commencing operation
krb5kdc: starting...
W20260929 17:52:04.359846 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:04 dist-test-slave-wbpt krb5kdc[25069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:52:04.487504 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:04.601775 25082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:04.602084 25082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:04.602129 25082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:04.606372 25082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:04.606460 25082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:04.606479 25082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:04.606494 25082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:04.606508 25082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:04.612201 25082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:04.613351 25082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:04.614408 25082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:04.623340 25087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:04.623383 25082 server_base.cc:1034] running on GCE node
W20260929 17:52:04.623458 25088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:04.623332 25090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:04.624316 25082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:04.625474 25082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:04.626662 25082 hybrid_clock.cc:648] HybridClock initialized: now 1790704324626640 us; error 36 us; skew 500 ppm
Sep 29 17:52:04 dist-test-slave-wbpt krb5kdc[25069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:04.635031 25082 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:52:04.636107 25082 webserver.cc:460] Webserver started at http://127.0.0.1:33275/ using document root <none> and password file <none>
I20260929 17:52:04.636654 25082 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:04.636713 25082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:04.636897 25082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:04.638990 25082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "af320d5dbbe640fdab7ae64c7540d939"
format_stamp: "Formatted at 2026-09-29 17:52:04 on dist-test-slave-wbpt"
server_key: "bdfb01fc0f2e883789e655fce329d15f"
server_key_iv: "f76631e8a7f777b532abf90012222c5a"
server_key_version: "encryptionkey@0"
I20260929 17:52:04.639506 25082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "af320d5dbbe640fdab7ae64c7540d939"
format_stamp: "Formatted at 2026-09-29 17:52:04 on dist-test-slave-wbpt"
server_key: "bdfb01fc0f2e883789e655fce329d15f"
server_key_iv: "f76631e8a7f777b532abf90012222c5a"
server_key_version: "encryptionkey@0"
I20260929 17:52:04.643412 25082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:52:04.646032 25097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:04.647389 25082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:52:04.647549 25082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "af320d5dbbe640fdab7ae64c7540d939"
format_stamp: "Formatted at 2026-09-29 17:52:04 on dist-test-slave-wbpt"
server_key: "bdfb01fc0f2e883789e655fce329d15f"
server_key_iv: "f76631e8a7f777b532abf90012222c5a"
server_key_version: "encryptionkey@0"
I20260929 17:52:04.647675 25082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:04.657673 25082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:04.660907 25082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:04.661092 25082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:04.667927 25082 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36565
I20260929 17:52:04.667965 25149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36565 every 8 connection(s)
I20260929 17:52:04.668918 25082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:04.671772 25150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:04.674382 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25082
I20260929 17:52:04.674497 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:04.674755 3807 external_mini_cluster.cc:1428] Setting key 97d12bd62504a21da3cc7fd6c903fb75
I20260929 17:52:04.677964 25150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939: Bootstrap starting.
I20260929 17:52:04.680365 25150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:04.681130 25150 log.cc:824] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:04.683542 25150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939: No bootstrap required, opened a new log
Sep 29 17:52:04 dist-test-slave-wbpt krb5kdc[25069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:52:04.687078 25150 raft_consensus.cc:348] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af320d5dbbe640fdab7ae64c7540d939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260929 17:52:04.687325 25150 raft_consensus.cc:374] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:04.687358 25150 raft_consensus.cc:729] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af320d5dbbe640fdab7ae64c7540d939, State: Initialized, Role: FOLLOWER
I20260929 17:52:04.687904 25150 consensus_queue.cc:260] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af320d5dbbe640fdab7ae64c7540d939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260929 17:52:04.688258 25150 raft_consensus.cc:388] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:04.688408 25150 raft_consensus.cc:482] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:04.688539 25150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:04.689989 25150 raft_consensus.cc:504] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af320d5dbbe640fdab7ae64c7540d939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260929 17:52:04.690438 25150 leader_election.cc:302] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af320d5dbbe640fdab7ae64c7540d939; no voters:
I20260929 17:52:04.690850 25150 leader_election.cc:288] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:04.690968 25155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:04.691025 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:04.676217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:46497 (local address 127.0.0.1:36565)
0929 17:52:04.676701 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:04.676716 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:04.676754 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:52:04.677411 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:04.677429 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:04.677808 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:04.678131 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:04.678956 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:04.680020 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:04.681058 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:04.681619 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:04.681787 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:04.684928 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:04.684939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:04.684958 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:04.685032 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:04.687245 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:04.687876 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:04.687882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:04.687886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:04.688035 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:04.688415 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:04.688420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:04.688423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:04.689044 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:04.689306 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:04.689560 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 17:52:04.689862 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260929 17:52:04.691174 25155 raft_consensus.cc:686] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [term 1 LEADER]: Becoming Leader. State: Replica: af320d5dbbe640fdab7ae64c7540d939, State: Running, Role: LEADER
I20260929 17:52:04.691511 25155 consensus_queue.cc:237] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af320d5dbbe640fdab7ae64c7540d939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260929 17:52:04.692317 25150 sys_catalog.cc:564] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:04.693226 25157 sys_catalog.cc:455] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af320d5dbbe640fdab7ae64c7540d939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af320d5dbbe640fdab7ae64c7540d939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } } }
I20260929 17:52:04.693351 25157 sys_catalog.cc:458] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:04.693765 25164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:04.693742 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af320d5dbbe640fdab7ae64c7540d939. Latest consensus state: current_term: 1 leader_uuid: "af320d5dbbe640fdab7ae64c7540d939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af320d5dbbe640fdab7ae64c7540d939" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } } }
I20260929 17:52:04.693923 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:04.694783 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:04.697165 25164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:04.703617 25164 catalog_manager.cc:1347] Generated new cluster ID: adf4c83335424154a35d9d39a3d44d90
I20260929 17:52:04.703702 25164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:04.707710 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:04.699311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:49377 (local address 127.0.0.1:36565)
0929 17:52:04.699425 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:04.699428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:04.699857 (+ 429us) server_negotiation.cc:407] Connection header received
0929 17:52:04.700011 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:04.700015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:04.700060 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:04.700123 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:04.700813 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:04.701333 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:04.702289 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:04.702654 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:04.702778 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:04.704408 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:04.704412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:04.704415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:04.704440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:04.705961 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:04.706476 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:04.706480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:04.706482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:04.706563 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:04.706866 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:04.706869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:04.706871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:04.707050 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:04.707194 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:04.707453 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 17:52:04.707545 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 17:52:04.713747 25164 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:04.715090 25164 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:04.722684 25164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af320d5dbbe640fdab7ae64c7540d939: Generated new TSK 0
I20260929 17:52:04.723517 25164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:04.728437 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25082
W20260929 17:52:04.733803 25067 connection.cc:535] client connection to 127.0.0.1:36565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36565): Connection reset by peer (error 104)
2026-09-29T17:52:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:04 dist-test-slave-wbpt krb5kdc[25181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:04 dist-test-slave-wbpt krb5kdc[25181](info): set up 2 sockets
Sep 29 17:52:04 dist-test-slave-wbpt krb5kdc[25181](info): commencing operation
krb5kdc: starting...
W20260929 17:52:05.692147 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:05 dist-test-slave-wbpt krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:52:05.813094 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:05.924291 25194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:05.924590 25194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:05.924633 25194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:05.928779 25194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:05.928848 25194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:05.928866 25194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:05.928881 25194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:05.928895 25194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:05.934175 25194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:05.935230 25194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:05.936301 25194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:05.944659 25200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:05.944598 25199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:05.944568 25202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:05.944890 25194 server_base.cc:1034] running on GCE node
I20260929 17:52:05.945505 25194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:05.946609 25194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:05.947763 25194 hybrid_clock.cc:648] HybridClock initialized: now 1790704325947743 us; error 32 us; skew 500 ppm
Sep 29 17:52:05 dist-test-slave-wbpt krb5kdc[25181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:05.950493 25194 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:52:05.951350 25194 webserver.cc:460] Webserver started at http://127.0.0.1:44965/ using document root <none> and password file <none>
I20260929 17:52:05.951921 25194 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:05.951992 25194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:05.952174 25194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:05.954392 25194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "32bd4503e60b4d828907c03275216b33"
format_stamp: "Formatted at 2026-09-29 17:52:05 on dist-test-slave-wbpt"
server_key: "028dae953e94b61d9756287dddf0cf9c"
server_key_iv: "e8d4fb8d3da0263714eac57529f1e9c1"
server_key_version: "encryptionkey@0"
I20260929 17:52:05.954963 25194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "32bd4503e60b4d828907c03275216b33"
format_stamp: "Formatted at 2026-09-29 17:52:05 on dist-test-slave-wbpt"
server_key: "028dae953e94b61d9756287dddf0cf9c"
server_key_iv: "e8d4fb8d3da0263714eac57529f1e9c1"
server_key_version: "encryptionkey@0"
I20260929 17:52:05.959049 25194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 17:52:05.961614 25209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:05.962930 25194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:52:05.963055 25194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "32bd4503e60b4d828907c03275216b33"
format_stamp: "Formatted at 2026-09-29 17:52:05 on dist-test-slave-wbpt"
server_key: "028dae953e94b61d9756287dddf0cf9c"
server_key_iv: "e8d4fb8d3da0263714eac57529f1e9c1"
server_key_version: "encryptionkey@0"
I20260929 17:52:05.963153 25194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:05.974341 25194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:05.977538 25194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:05.977722 25194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:05.984697 25194 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42761
I20260929 17:52:05.984735 25261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42761 every 8 connection(s)
I20260929 17:52:05.985700 25194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:05.988580 25262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:05.989310 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25194
I20260929 17:52:05.989446 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:05.989729 3807 external_mini_cluster.cc:1428] Setting key 28a784bf14be9c37bd7c0257f7dae5b6
I20260929 17:52:05.995774 25262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33: Bootstrap starting.
I20260929 17:52:05.998612 25262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:05.999544 25262 log.cc:824] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:06.000196 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:05.991055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:49475 (local address 127.0.0.1:42761)
0929 17:52:05.991486 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:05.991510 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:05.991562 (+ 52us) server_negotiation.cc:407] Connection header received
0929 17:52:05.992417 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:05.992445 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:05.992937 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:05.993332 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:05.993789 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:05.995295 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:05.996375 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:05.997084 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:05.997265 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:05.998016 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:05.998023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:05.998040 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:05.998559 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:05.998633 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:05.998664 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:05.998868 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:05.999031 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 17:52:05.999172 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260929 17:52:06.001935 25262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33: No bootstrap required, opened a new log
I20260929 17:52:06.003420 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:06.006772 25262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd4503e60b4d828907c03275216b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42761 } }
I20260929 17:52:06.007169 25262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:06.007247 25262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bd4503e60b4d828907c03275216b33, State: Initialized, Role: FOLLOWER
I20260929 17:52:06.008170 25262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd4503e60b4d828907c03275216b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42761 } }
I20260929 17:52:06.008406 25262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:06.008478 25262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:06.008594 25262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:06.009732 25262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd4503e60b4d828907c03275216b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42761 } }
I20260929 17:52:06.010151 25262 leader_election.cc:302] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32bd4503e60b4d828907c03275216b33; no voters:
I20260929 17:52:06.010528 25262 leader_election.cc:288] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:06.010654 25272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:06.010869 25272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [term 1 LEADER]: Becoming Leader. State: Replica: 32bd4503e60b4d828907c03275216b33, State: Running, Role: LEADER
I20260929 17:52:06.011232 25272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd4503e60b4d828907c03275216b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42761 } }
I20260929 17:52:06.011525 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:06.006109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:56959 (local address 127.0.0.1:42761)
0929 17:52:06.006246 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:06.006251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:06.006646 (+ 395us) server_negotiation.cc:407] Connection header received
0929 17:52:06.006851 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:06.006855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:06.006909 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:06.006997 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:06.007474 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:06.008505 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:06.009415 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:06.009766 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:06.009885 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:06.010984 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:06.010988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:06.010991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:06.011006 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:06.011038 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:06.011051 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:06.011111 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:06.011346 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 17:52:06.011389 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:52:06.011987 25262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:06.013155 25273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32bd4503e60b4d828907c03275216b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd4503e60b4d828907c03275216b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42761 } } }
I20260929 17:52:06.013155 25274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32bd4503e60b4d828907c03275216b33. Latest consensus state: current_term: 1 leader_uuid: "32bd4503e60b4d828907c03275216b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd4503e60b4d828907c03275216b33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42761 } } }
I20260929 17:52:06.013330 25273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:06.013330 25274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:06.013731 25281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:06.017279 25281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:06.025341 25281 catalog_manager.cc:1347] Generated new cluster ID: 96fa0f5848084ba4891a075a66b8a478
I20260929 17:52:06.025532 25281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:06.040603 25281 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:06.042042 25281 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:06.059499 25281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32bd4503e60b4d828907c03275216b33: Generated new TSK 0
I20260929 17:52:06.060495 25281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:06.068328 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25194
W20260929 17:52:06.073374 25179 connection.cc:535] client connection to 127.0.0.1:42761 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42761): Connection reset by peer (error 104)
2026-09-29T17:52:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:06 dist-test-slave-wbpt krb5kdc[25293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:06 dist-test-slave-wbpt krb5kdc[25293](info): set up 2 sockets
Sep 29 17:52:06 dist-test-slave-wbpt krb5kdc[25293](info): commencing operation
krb5kdc: starting...
W20260929 17:52:07.036029 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:07 dist-test-slave-wbpt krb5kdc[25293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:52:07.163898 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:07.277462 25306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:07.277737 25306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:07.277781 25306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:07.281888 25306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:07.281955 25306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:07.281975 25306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:07.281991 25306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:07.282004 25306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:07.287509 25306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:07.288702 25306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:07.289733 25306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:07.298481 25314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:07.298467 25312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:07.298475 25311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:07.298772 25306 server_base.cc:1034] running on GCE node
I20260929 17:52:07.299454 25306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:07.300740 25306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:07.301962 25306 hybrid_clock.cc:648] HybridClock initialized: now 1790704327301914 us; error 59 us; skew 500 ppm
Sep 29 17:52:07 dist-test-slave-wbpt krb5kdc[25293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:07.304831 25306 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:52:07.305779 25306 webserver.cc:460] Webserver started at http://127.0.0.1:33075/ using document root <none> and password file <none>
I20260929 17:52:07.306324 25306 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:07.306389 25306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:07.306587 25306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:07.308691 25306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "0264e1ae50354220a119f91f3fa01baa"
format_stamp: "Formatted at 2026-09-29 17:52:07 on dist-test-slave-wbpt"
server_key: "8d4124d988c69c0fdcc4e5487b823bd9"
server_key_iv: "ddf93261a3773962e9ca554b2e79197b"
server_key_version: "encryptionkey@0"
I20260929 17:52:07.309216 25306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "0264e1ae50354220a119f91f3fa01baa"
format_stamp: "Formatted at 2026-09-29 17:52:07 on dist-test-slave-wbpt"
server_key: "8d4124d988c69c0fdcc4e5487b823bd9"
server_key_iv: "ddf93261a3773962e9ca554b2e79197b"
server_key_version: "encryptionkey@0"
I20260929 17:52:07.312879 25306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:52:07.315321 25321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:07.316771 25306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:52:07.316898 25306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "0264e1ae50354220a119f91f3fa01baa"
format_stamp: "Formatted at 2026-09-29 17:52:07 on dist-test-slave-wbpt"
server_key: "8d4124d988c69c0fdcc4e5487b823bd9"
server_key_iv: "ddf93261a3773962e9ca554b2e79197b"
server_key_version: "encryptionkey@0"
I20260929 17:52:07.317010 25306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:07.321522 25306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:07.324520 25306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:07.324694 25306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:07.331727 25306 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36877
I20260929 17:52:07.331839 25373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36877 every 8 connection(s)
I20260929 17:52:07.332742 25306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:07.335549 25374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:07.340770 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25306
I20260929 17:52:07.340909 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:07.341162 3807 external_mini_cluster.cc:1428] Setting key a76b0ef3a2ecb625f6eecf6251a811f3
I20260929 17:52:07.341872 25374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa: Bootstrap starting.
I20260929 17:52:07.344764 25374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:07.345705 25374 log.cc:824] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:07.347380 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:07.342535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:33297 (local address 127.0.0.1:36877)
0929 17:52:07.343010 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:07.343037 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:07.343085 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:52:07.344048 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:07.344074 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:07.344468 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:07.345244 (+ 776us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:07.345920 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:07.345925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:07.345941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:07.345951 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:07.346011 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:07.346039 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:07.346152 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:07.346275 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 17:52:07.346438 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260929 17:52:07.348191 25374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa: No bootstrap required, opened a new log
I20260929 17:52:07.350335 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:07.352035 25374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0264e1ae50354220a119f91f3fa01baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260929 17:52:07.352300 25374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:07.352358 25374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0264e1ae50354220a119f91f3fa01baa, State: Initialized, Role: FOLLOWER
I20260929 17:52:07.352809 25374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0264e1ae50354220a119f91f3fa01baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260929 17:52:07.352990 25374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:07.353070 25374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:07.353168 25374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:07.353927 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:07.352297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:34719 (local address 127.0.0.1:36877)
0929 17:52:07.352408 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:07.352412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:07.352643 (+ 231us) server_negotiation.cc:407] Connection header received
0929 17:52:07.352748 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:07.352751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:07.352788 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:07.352923 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:07.353595 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:07.353598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:07.353601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:07.353613 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:07.353640 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:07.353653 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:07.353734 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:07.353778 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 17:52:07.353815 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:52:07.354277 25374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0264e1ae50354220a119f91f3fa01baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260929 17:52:07.354632 25374 leader_election.cc:302] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0264e1ae50354220a119f91f3fa01baa; no voters:
I20260929 17:52:07.354990 25374 leader_election.cc:288] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:07.355053 25384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:07.355257 25384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [term 1 LEADER]: Becoming Leader. State: Replica: 0264e1ae50354220a119f91f3fa01baa, State: Running, Role: LEADER
I20260929 17:52:07.355566 25384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0264e1ae50354220a119f91f3fa01baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260929 17:52:07.356242 25374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:07.357292 25386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0264e1ae50354220a119f91f3fa01baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0264e1ae50354220a119f91f3fa01baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } } }
I20260929 17:52:07.357429 25386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:07.357650 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0264e1ae50354220a119f91f3fa01baa. Latest consensus state: current_term: 1 leader_uuid: "0264e1ae50354220a119f91f3fa01baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0264e1ae50354220a119f91f3fa01baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } } }
I20260929 17:52:07.357738 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:07.357877 25393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:07.360983 25393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:07.367151 25393 catalog_manager.cc:1347] Generated new cluster ID: 6fbe671a9f594402957f7cf9508842fb
I20260929 17:52:07.367231 25393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:07.383431 25393 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:07.383935 25393 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:07.390297 25393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0264e1ae50354220a119f91f3fa01baa: Generated new TSK 0
I20260929 17:52:07.391151 25393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:07.399686 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25306
2026-09-29T17:52:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:07 dist-test-slave-wbpt krb5kdc[25405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:07 dist-test-slave-wbpt krb5kdc[25405](info): set up 2 sockets
Sep 29 17:52:07 dist-test-slave-wbpt krb5kdc[25405](info): commencing operation
krb5kdc: starting...
W20260929 17:52:08.355707 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:08 dist-test-slave-wbpt krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:08.478367 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44695
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:08.589021 25418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:08.589298 25418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:08.589340 25418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:08.593438 25418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:08.593524 25418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:08.593545 25418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:08.593562 25418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:08.593583 25418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:08.599174 25418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:44695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:08.600320 25418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:08.601308 25418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:08.609736 25424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:08.609746 25423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:08.609751 25426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:08.610014 25418 server_base.cc:1034] running on GCE node
I20260929 17:52:08.610656 25418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:08.611836 25418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:08.613034 25418 hybrid_clock.cc:648] HybridClock initialized: now 1790704328613013 us; error 40 us; skew 500 ppm
Sep 29 17:52:08 dist-test-slave-wbpt krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704328, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:08.616072 25418 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:08.617105 25418 webserver.cc:460] Webserver started at http://127.3.183.254:35847/ using document root <none> and password file <none>
I20260929 17:52:08.617676 25418 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:08.617743 25418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:08.617952 25418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:08.620194 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e"
format_stamp: "Formatted at 2026-09-29 17:52:08 on dist-test-slave-wbpt"
server_key: "6a5d02bcff1b3080cf0010b9673be77d"
server_key_iv: "0d6dc595cd541d367a4f294a1ea933fb"
server_key_version: "encryptionkey@0"
I20260929 17:52:08.620790 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e"
format_stamp: "Formatted at 2026-09-29 17:52:08 on dist-test-slave-wbpt"
server_key: "6a5d02bcff1b3080cf0010b9673be77d"
server_key_iv: "0d6dc595cd541d367a4f294a1ea933fb"
server_key_version: "encryptionkey@0"
I20260929 17:52:08.624830 25418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:52:08.627331 25433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:08.628667 25418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:08.628798 25418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e"
format_stamp: "Formatted at 2026-09-29 17:52:08 on dist-test-slave-wbpt"
server_key: "6a5d02bcff1b3080cf0010b9673be77d"
server_key_iv: "0d6dc595cd541d367a4f294a1ea933fb"
server_key_version: "encryptionkey@0"
I20260929 17:52:08.628903 25418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:08.657146 25418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:08.660653 25418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:08.660878 25418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:08.668314 25418 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:44695
I20260929 17:52:08.668322 25485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:44695 every 8 connection(s)
I20260929 17:52:08.669373 25418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:08.672240 25486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:08.675102 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25418
I20260929 17:52:08.675252 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:08.675544 3807 external_mini_cluster.cc:1428] Setting key 40772896d5311aaae52a3a934d11cd57
I20260929 17:52:08.678438 25486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e: Bootstrap starting.
I20260929 17:52:08.681141 25486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:08.682060 25486 log.cc:824] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:08.684454 25486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e: No bootstrap required, opened a new log
Sep 29 17:52:08 dist-test-slave-wbpt krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:08.687462 25486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44695 } }
I20260929 17:52:08.687716 25486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:08.687798 25486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b01b4b0dcf249efbf7ca2770eb74a2e, State: Initialized, Role: FOLLOWER
I20260929 17:52:08.688268 25486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44695 } }
I20260929 17:52:08.688409 25486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:08.688474 25486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:08.688577 25486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:08.689535 25486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44695 } }
I20260929 17:52:08.689894 25486 leader_election.cc:302] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b01b4b0dcf249efbf7ca2770eb74a2e; no voters:
I20260929 17:52:08.690219 25486 leader_election.cc:288] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:08.690333 25491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:08.690496 25491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [term 1 LEADER]: Becoming Leader. State: Replica: 4b01b4b0dcf249efbf7ca2770eb74a2e, State: Running, Role: LEADER
I20260929 17:52:08.690691 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:08.676968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45971 (local address 127.3.183.254:44695)
0929 17:52:08.677399 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:08.677417 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:08.677452 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:52:08.678077 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:08.678094 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:08.678452 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:08.678768 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:08.679628 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:08.680807 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:08.681991 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:08.682548 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:08.682700 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:08.685423 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:08.685428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:08.685442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:08.685471 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:08.687547 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:08.688141 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:08.688146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:08.688150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:08.688264 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:08.688582 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:08.688585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:08.688587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:08.689022 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:08.689235 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:08.689551 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 17:52:08.689787 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
I20260929 17:52:08.690820 25491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44695 } }
I20260929 17:52:08.691547 25486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:08.692093 25493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b01b4b0dcf249efbf7ca2770eb74a2e. Latest consensus state: current_term: 1 leader_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44695 } } }
I20260929 17:52:08.692289 25493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:08.692849 25492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b01b4b0dcf249efbf7ca2770eb74a2e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 44695 } } }
I20260929 17:52:08.692939 25492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:08.693152 25498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:08.694092 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:08.696911 25498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:08.704370 25498 catalog_manager.cc:1347] Generated new cluster ID: bc10e711e4a546be83611f98d9da3014
I20260929 17:52:08.704454 25498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:08.705613 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:08.696484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57565 (local address 127.3.183.254:44695)
0929 17:52:08.696604 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:08.696608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:08.696927 (+ 319us) server_negotiation.cc:407] Connection header received
0929 17:52:08.697172 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:08.697175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:08.697227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:08.697279 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:08.698124 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:08.698698 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:08.699773 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:08.700276 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:08.700403 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:08.701939 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:08.701943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:08.701946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:08.701974 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:08.703639 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:08.704253 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:08.704257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:08.704259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:08.704365 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:08.704747 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:08.704751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:08.704753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:08.704951 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:08.705083 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:08.705356 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 17:52:08.705455 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:52:08.720363 25498 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:08.721382 25498 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:08.727422 25498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b01b4b0dcf249efbf7ca2770eb74a2e: Generated new TSK 0
I20260929 17:52:08.728214 25498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:08.738134 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25418
W20260929 17:52:08.743196 25403 connection.cc:535] client connection to 127.3.183.254:44695 recv error: Network error: failed to read from TLS socket (remote: 127.3.183.254:44695): Connection reset by peer (error 104)
2026-09-29T17:52:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:08 dist-test-slave-wbpt krb5kdc[25517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:08 dist-test-slave-wbpt krb5kdc[25517](info): set up 2 sockets
Sep 29 17:52:08 dist-test-slave-wbpt krb5kdc[25517](info): commencing operation
krb5kdc: starting...
W20260929 17:52:09.707890 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:09 dist-test-slave-wbpt krb5kdc[25517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:09.835276 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:43333
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:43333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:09.946393 25530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:09.946668 25530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:09.946708 25530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:09.950649 25530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:09.950703 25530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:09.950721 25530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:09.950737 25530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:09.950752 25530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:09.956022 25530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:43333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:09.957136 25530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:09.958173 25530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:09.966355 25535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:09.966389 25536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:09.966542 25530 server_base.cc:1034] running on GCE node
W20260929 17:52:09.966355 25538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:09.967232 25530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:09.968375 25530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:09.969540 25530 hybrid_clock.cc:648] HybridClock initialized: now 1790704329969517 us; error 34 us; skew 500 ppm
Sep 29 17:52:09 dist-test-slave-wbpt krb5kdc[25517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704329, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:09.980472 25530 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:09.981490 25530 webserver.cc:460] Webserver started at http://127.3.183.254:46441/ using document root <none> and password file <none>
I20260929 17:52:09.982089 25530 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:09.982160 25530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:09.982354 25530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:09.984614 25530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "1f25cc2de60c4b9a9525d6afb35f54c4"
format_stamp: "Formatted at 2026-09-29 17:52:09 on dist-test-slave-wbpt"
server_key: "18dc9f4adc0b932c09eb56506837ca1b"
server_key_iv: "3551f5f80161cb35218a2b9e920c3d3a"
server_key_version: "encryptionkey@0"
I20260929 17:52:09.985177 25530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "1f25cc2de60c4b9a9525d6afb35f54c4"
format_stamp: "Formatted at 2026-09-29 17:52:09 on dist-test-slave-wbpt"
server_key: "18dc9f4adc0b932c09eb56506837ca1b"
server_key_iv: "3551f5f80161cb35218a2b9e920c3d3a"
server_key_version: "encryptionkey@0"
I20260929 17:52:09.989001 25530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 17:52:09.991645 25545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:09.993046 25530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:52:09.993186 25530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "1f25cc2de60c4b9a9525d6afb35f54c4"
format_stamp: "Formatted at 2026-09-29 17:52:09 on dist-test-slave-wbpt"
server_key: "18dc9f4adc0b932c09eb56506837ca1b"
server_key_iv: "3551f5f80161cb35218a2b9e920c3d3a"
server_key_version: "encryptionkey@0"
I20260929 17:52:09.993304 25530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:10.008363 25530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:10.011437 25530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:10.011615 25530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:10.018410 25530 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:43333
I20260929 17:52:10.018448 25597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:43333 every 8 connection(s)
I20260929 17:52:10.019352 25530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:10.021704 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25530
I20260929 17:52:10.021875 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:10.022149 3807 external_mini_cluster.cc:1428] Setting key 32f6b560f621b90623c17c7a421de031
I20260929 17:52:10.022414 25598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:10.029613 25598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4: Bootstrap starting.
I20260929 17:52:10.032310 25598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:10.032770 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:10.023569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37433 (local address 127.3.183.254:43333)
0929 17:52:10.024006 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:10.024024 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:10.024066 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:52:10.025064 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:10.025088 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:10.025551 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:10.025925 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:10.026348 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:10.027842 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:10.028806 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:10.029500 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:10.029654 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:10.030674 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:10.030681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:10.030698 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:10.031203 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:10.031270 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:10.031300 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:10.031434 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:10.031621 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 17:52:10.031755 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":145,"threads_started":1}
I20260929 17:52:10.033169 25598 log.cc:824] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:10.035446 25598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4: No bootstrap required, opened a new log
I20260929 17:52:10.036227 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:10.039438 25598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43333 } }
I20260929 17:52:10.039716 25598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:10.039803 25598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f25cc2de60c4b9a9525d6afb35f54c4, State: Initialized, Role: FOLLOWER
I20260929 17:52:10.040233 25598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43333 } }
I20260929 17:52:10.040364 25598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:10.040432 25598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:10.040529 25598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:10.041590 25598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43333 } }
I20260929 17:52:10.041937 25598 leader_election.cc:302] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f25cc2de60c4b9a9525d6afb35f54c4; no voters:
I20260929 17:52:10.042120 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:10.038166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55523 (local address 127.3.183.254:43333)
0929 17:52:10.038294 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:10.038299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:10.038484 (+ 185us) server_negotiation.cc:407] Connection header received
0929 17:52:10.038585 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:10.038588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:10.038632 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:10.038695 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:10.038953 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:10.039445 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:10.040396 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:10.040698 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:10.040788 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:10.041651 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:10.041655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:10.041658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:10.041672 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:10.041701 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:10.041714 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:10.041794 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:10.041945 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 17:52:10.041984 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:52:10.042373 25598 leader_election.cc:288] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:10.043002 25608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:10.043238 25608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [term 1 LEADER]: Becoming Leader. State: Replica: 1f25cc2de60c4b9a9525d6afb35f54c4, State: Running, Role: LEADER
I20260929 17:52:10.043609 25608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43333 } }
I20260929 17:52:10.043795 25598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:10.045403 25610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f25cc2de60c4b9a9525d6afb35f54c4. Latest consensus state: current_term: 1 leader_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43333 } } }
I20260929 17:52:10.045553 25610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:10.045848 25609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f25cc2de60c4b9a9525d6afb35f54c4" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 43333 } } }
I20260929 17:52:10.046271 25609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4 [sys.catalog]: This master's current role is: LEADER
W20260929 17:52:10.048542 25623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:10.048612 25623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:52:10.048723 25622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:10.049625 25622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:10.055646 25622 catalog_manager.cc:1347] Generated new cluster ID: bfb4e5ca01784c6ca7e49f11fd1b4a10
I20260929 17:52:10.055742 25622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:10.077867 25622 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:10.078809 25622 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:10.090683 25622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f25cc2de60c4b9a9525d6afb35f54c4: Generated new TSK 0
I20260929 17:52:10.091585 25622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:10.108081 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25530
W20260929 17:52:10.113178 25515 connection.cc:535] client connection to 127.3.183.254:43333 recv error: Network error: failed to read from TLS socket (remote: 127.3.183.254:43333): Connection reset by peer (error 104)
2026-09-29T17:52:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:10 dist-test-slave-wbpt krb5kdc[25629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:10 dist-test-slave-wbpt krb5kdc[25629](info): set up 2 sockets
Sep 29 17:52:10 dist-test-slave-wbpt krb5kdc[25629](info): commencing operation
krb5kdc: starting...
W20260929 17:52:11.059851 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:11 dist-test-slave-wbpt krb5kdc[25629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:11.179315 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:40829
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:11.289351 25642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:11.289594 25642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:11.289633 25642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:11.293509 25642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:11.293557 25642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:11.293576 25642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:11.293591 25642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:11.293605 25642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:11.298755 25642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:40829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:11.299693 25642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:11.300674 25642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:11.308662 25648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:11.308735 25642 server_base.cc:1034] running on GCE node
W20260929 17:52:11.308679 25650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:11.308686 25647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:11.309410 25642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:11.310446 25642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:11.311594 25642 hybrid_clock.cc:648] HybridClock initialized: now 1790704331311571 us; error 35 us; skew 500 ppm
Sep 29 17:52:11 dist-test-slave-wbpt krb5kdc[25629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704331, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:11.314357 25642 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:11.315264 25642 webserver.cc:460] Webserver started at http://127.3.183.254:42787/ using document root <none> and password file <none>
I20260929 17:52:11.315846 25642 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:11.315914 25642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:11.316104 25642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:11.318109 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "b71fad43f7d548e6af471adcc2bb855f"
format_stamp: "Formatted at 2026-09-29 17:52:11 on dist-test-slave-wbpt"
server_key: "a438dd588c8fcc19f803bfaa98b9fb5c"
server_key_iv: "0b0371c4a592ac474697b95a759c6f7d"
server_key_version: "encryptionkey@0"
I20260929 17:52:11.318656 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "b71fad43f7d548e6af471adcc2bb855f"
format_stamp: "Formatted at 2026-09-29 17:52:11 on dist-test-slave-wbpt"
server_key: "a438dd588c8fcc19f803bfaa98b9fb5c"
server_key_iv: "0b0371c4a592ac474697b95a759c6f7d"
server_key_version: "encryptionkey@0"
I20260929 17:52:11.322372 25642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:52:11.324807 25657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:11.326012 25642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:11.326128 25642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "b71fad43f7d548e6af471adcc2bb855f"
format_stamp: "Formatted at 2026-09-29 17:52:11 on dist-test-slave-wbpt"
server_key: "a438dd588c8fcc19f803bfaa98b9fb5c"
server_key_iv: "0b0371c4a592ac474697b95a759c6f7d"
server_key_version: "encryptionkey@0"
I20260929 17:52:11.326225 25642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:11.330549 25642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:11.333515 25642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:11.333674 25642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:11.340551 25642 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:40829
I20260929 17:52:11.340569 25709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:40829 every 8 connection(s)
I20260929 17:52:11.341472 25642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:11.344292 25710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:11.345484 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25642
I20260929 17:52:11.345643 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:11.345948 3807 external_mini_cluster.cc:1428] Setting key 8e12f772a6a5e633d2299580b293d176
I20260929 17:52:11.351044 25710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f: Bootstrap starting.
I20260929 17:52:11.351814 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:11.347438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60275 (local address 127.3.183.254:40829)
0929 17:52:11.347905 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:11.347925 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:11.347967 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:52:11.348790 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:11.348815 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:11.349204 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:11.349592 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:11.350345 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:11.350350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:11.350365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:11.350377 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:11.350448 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:11.350473 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:11.350609 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:11.350735 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 17:52:11.350888 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260929 17:52:11.353389 25710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:11.354157 25710 log.cc:824] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:11.354522 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:11.356184 25710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f: No bootstrap required, opened a new log
I20260929 17:52:11.358132 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:11.356534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37627 (local address 127.3.183.254:40829)
0929 17:52:11.356654 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:11.356658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:11.356871 (+ 213us) server_negotiation.cc:407] Connection header received
0929 17:52:11.356971 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:11.356974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:11.357008 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:11.357106 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:11.357799 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:11.357803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:11.357806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:11.357817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:11.357843 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:11.357855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:11.357946 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:11.357979 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 17:52:11.358015 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:52:11.360169 25710 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71fad43f7d548e6af471adcc2bb855f" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40829 } }
I20260929 17:52:11.360389 25710 raft_consensus.cc:374] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:11.360437 25710 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71fad43f7d548e6af471adcc2bb855f, State: Initialized, Role: FOLLOWER
I20260929 17:52:11.360826 25710 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71fad43f7d548e6af471adcc2bb855f" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40829 } }
I20260929 17:52:11.360951 25710 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:11.361012 25710 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:11.361104 25710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:11.362119 25710 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71fad43f7d548e6af471adcc2bb855f" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40829 } }
I20260929 17:52:11.362435 25710 leader_election.cc:302] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71fad43f7d548e6af471adcc2bb855f; no voters:
I20260929 17:52:11.362718 25710 leader_election.cc:288] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:11.362841 25720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:11.363080 25720 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [term 1 LEADER]: Becoming Leader. State: Replica: b71fad43f7d548e6af471adcc2bb855f, State: Running, Role: LEADER
I20260929 17:52:11.363437 25720 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71fad43f7d548e6af471adcc2bb855f" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40829 } }
I20260929 17:52:11.363914 25710 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:11.365370 25721 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b71fad43f7d548e6af471adcc2bb855f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71fad43f7d548e6af471adcc2bb855f" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40829 } } }
I20260929 17:52:11.365501 25721 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:11.365785 25722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71fad43f7d548e6af471adcc2bb855f. Latest consensus state: current_term: 1 leader_uuid: "b71fad43f7d548e6af471adcc2bb855f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71fad43f7d548e6af471adcc2bb855f" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 40829 } } }
I20260929 17:52:11.365895 25734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:11.366225 25722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:11.368834 25734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:11.374187 25734 catalog_manager.cc:1347] Generated new cluster ID: 8937d4bb59a84262a638f0a55578d8c3
I20260929 17:52:11.374269 25734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:11.386003 25734 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:11.386372 25734 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:11.395996 25734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b71fad43f7d548e6af471adcc2bb855f: Generated new TSK 0
I20260929 17:52:11.396844 25734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:11.409612 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25642
2026-09-29T17:52:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:11 dist-test-slave-wbpt krb5kdc[25741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:11 dist-test-slave-wbpt krb5kdc[25741](info): set up 2 sockets
Sep 29 17:52:11 dist-test-slave-wbpt krb5kdc[25741](info): commencing operation
krb5kdc: starting...
W20260929 17:52:12.367864 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:12 dist-test-slave-wbpt krb5kdc[25741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:12.484983 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:42241
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:12.594399 25754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:12.594683 25754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:12.594744 25754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:12.598834 25754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:12.598912 25754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:12.598938 25754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:12.598959 25754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:12.598980 25754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:12.604447 25754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:42241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:12.605551 25754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:12.606597 25754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:12.614972 25759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:12.615048 25754 server_base.cc:1034] running on GCE node
W20260929 17:52:12.615061 25760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:12.614981 25762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:12.615882 25754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:12.617064 25754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:12.618224 25754 hybrid_clock.cc:648] HybridClock initialized: now 1790704332618194 us; error 44 us; skew 500 ppm
Sep 29 17:52:12 dist-test-slave-wbpt krb5kdc[25741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:12.620995 25754 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:12.621922 25754 webserver.cc:460] Webserver started at http://127.3.183.254:35265/ using document root <none> and password file <none>
I20260929 17:52:12.622464 25754 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:12.622525 25754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:12.622692 25754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:12.624774 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "3b051c27caf040799d8a9350bfe48fff"
format_stamp: "Formatted at 2026-09-29 17:52:12 on dist-test-slave-wbpt"
server_key: "f0113c723182f9ca139ec5bb088dd62a"
server_key_iv: "3b6d9cd8096159d7bd4e3331a68c0d0c"
server_key_version: "encryptionkey@0"
I20260929 17:52:12.625288 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "3b051c27caf040799d8a9350bfe48fff"
format_stamp: "Formatted at 2026-09-29 17:52:12 on dist-test-slave-wbpt"
server_key: "f0113c723182f9ca139ec5bb088dd62a"
server_key_iv: "3b6d9cd8096159d7bd4e3331a68c0d0c"
server_key_version: "encryptionkey@0"
I20260929 17:52:12.628928 25754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 17:52:12.631407 25769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:12.632714 25754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:52:12.632820 25754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "3b051c27caf040799d8a9350bfe48fff"
format_stamp: "Formatted at 2026-09-29 17:52:12 on dist-test-slave-wbpt"
server_key: "f0113c723182f9ca139ec5bb088dd62a"
server_key_iv: "3b6d9cd8096159d7bd4e3331a68c0d0c"
server_key_version: "encryptionkey@0"
I20260929 17:52:12.632949 25754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:12.660279 25754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:12.663511 25754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:12.663693 25754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:12.670707 25754 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:42241
I20260929 17:52:12.670728 25821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:42241 every 8 connection(s)
I20260929 17:52:12.671671 25754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:12.674590 25822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:12.680552 25822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff: Bootstrap starting.
I20260929 17:52:12.681502 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25754
I20260929 17:52:12.681643 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:12.681974 3807 external_mini_cluster.cc:1428] Setting key da3b16581ba8d3e039b4ef9122a7fc00
I20260929 17:52:12.683584 25822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:12.684378 25822 log.cc:824] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:12.686295 25822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff: No bootstrap required, opened a new log
I20260929 17:52:12.689831 25822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b051c27caf040799d8a9350bfe48fff" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42241 } }
I20260929 17:52:12.690056 25822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:12.690092 25822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b051c27caf040799d8a9350bfe48fff, State: Initialized, Role: FOLLOWER
Sep 29 17:52:12 dist-test-slave-wbpt krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:12.690539 25822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b051c27caf040799d8a9350bfe48fff" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42241 } }
I20260929 17:52:12.690671 25822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:12.690733 25822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:12.690824 25822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:12.691957 25822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b051c27caf040799d8a9350bfe48fff" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42241 } }
I20260929 17:52:12.692351 25822 leader_election.cc:302] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b051c27caf040799d8a9350bfe48fff; no voters:
I20260929 17:52:12.692692 25822 leader_election.cc:288] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:12.692834 25827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:12.693058 25827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [term 1 LEADER]: Becoming Leader. State: Replica: 3b051c27caf040799d8a9350bfe48fff, State: Running, Role: LEADER
I20260929 17:52:12.693367 25827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b051c27caf040799d8a9350bfe48fff" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42241 } }
I20260929 17:52:12.694073 25822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:12.695067 25829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b051c27caf040799d8a9350bfe48fff. Latest consensus state: current_term: 1 leader_uuid: "3b051c27caf040799d8a9350bfe48fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b051c27caf040799d8a9350bfe48fff" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42241 } } }
I20260929 17:52:12.695192 25829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:12.695490 25828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b051c27caf040799d8a9350bfe48fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b051c27caf040799d8a9350bfe48fff" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 42241 } } }
I20260929 17:52:12.695601 25828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:12.695673 25836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:12.696895 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:12.683111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53845 (local address 127.3.183.254:42241)
0929 17:52:12.683529 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:12.683546 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:12.683583 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:52:12.684320 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:12.684341 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:12.684743 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:12.685072 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:12.685860 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:12.686879 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:12.687731 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:12.688262 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:12.688505 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:12.690938 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:12.690944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:12.690961 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:12.690991 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:12.692952 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:12.693459 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:12.693465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:12.693470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:12.693605 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:12.693932 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:12.693935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:12.693937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:12.694451 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:12.695377 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:12.695533 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 17:52:12.695893 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260929 17:52:12.699117 25836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:12.700080 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:12.704713 25836 catalog_manager.cc:1347] Generated new cluster ID: af865ae561564f539ccf35e0a39cb877
I20260929 17:52:12.704777 25836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:12.709592 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:12.702137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42001 (local address 127.3.183.254:42241)
0929 17:52:12.702235 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:12.702239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:12.702251 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:52:12.702285 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:12.702287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:12.702326 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:12.702389 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:12.703097 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:12.703927 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:12.704973 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:12.705231 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:12.705318 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:12.706600 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:12.706603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:12.706606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:12.706632 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:12.708000 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:12.708375 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:12.708378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:12.708380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:12.708455 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:12.708701 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:12.708704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:12.708706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:12.708877 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:12.709014 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:12.709223 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 17:52:12.709331 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:52:12.760841 25836 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:12.761868 25836 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:12.767395 25836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b051c27caf040799d8a9350bfe48fff: Generated new TSK 0
I20260929 17:52:12.768150 25836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:12.781148 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25754
2026-09-29T17:52:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:12 dist-test-slave-wbpt krb5kdc[25853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:12 dist-test-slave-wbpt krb5kdc[25853](info): set up 2 sockets
Sep 29 17:52:12 dist-test-slave-wbpt krb5kdc[25853](info): commencing operation
krb5kdc: starting...
W20260929 17:52:13.735932 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:13 dist-test-slave-wbpt krb5kdc[25853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:13.855542 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:37795
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:37795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:13.963869 25866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:13.964133 25866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:13.964193 25866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:13.968151 25866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:13.968207 25866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:13.968233 25866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:13.968266 25866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:13.968294 25866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:13.973877 25866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:37795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:37795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:13.974954 25866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:13.976016 25866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:13.984470 25872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:13.984575 25874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:13.984578 25871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:13.984694 25866 server_base.cc:1034] running on GCE node
I20260929 17:52:13.985368 25866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:13.986438 25866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:13.987604 25866 hybrid_clock.cc:648] HybridClock initialized: now 1790704333987589 us; error 25 us; skew 500 ppm
Sep 29 17:52:13 dist-test-slave-wbpt krb5kdc[25853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704333, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:13.990211 25866 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:13.991048 25866 webserver.cc:460] Webserver started at http://127.3.183.254:44077/ using document root <none> and password file <none>
I20260929 17:52:13.991549 25866 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:13.991601 25866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:13.991760 25866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:13.993770 25866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "1bdeeeb105854f2cbfdef74f610119a3"
format_stamp: "Formatted at 2026-09-29 17:52:13 on dist-test-slave-wbpt"
server_key: "b64aa6fe9f0a805c12b7de77bc662502"
server_key_iv: "453fca8fddaa813fc15f119fbd68b96f"
server_key_version: "encryptionkey@0"
I20260929 17:52:13.994271 25866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "1bdeeeb105854f2cbfdef74f610119a3"
format_stamp: "Formatted at 2026-09-29 17:52:13 on dist-test-slave-wbpt"
server_key: "b64aa6fe9f0a805c12b7de77bc662502"
server_key_iv: "453fca8fddaa813fc15f119fbd68b96f"
server_key_version: "encryptionkey@0"
I20260929 17:52:13.997855 25866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 17:52:14.000078 25881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:14.001399 25866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:52:14.001516 25866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "1bdeeeb105854f2cbfdef74f610119a3"
format_stamp: "Formatted at 2026-09-29 17:52:13 on dist-test-slave-wbpt"
server_key: "b64aa6fe9f0a805c12b7de77bc662502"
server_key_iv: "453fca8fddaa813fc15f119fbd68b96f"
server_key_version: "encryptionkey@0"
I20260929 17:52:14.001617 25866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:14.017828 25866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:14.020820 25866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:14.021005 25866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:14.027894 25866 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:37795
I20260929 17:52:14.027930 25933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:37795 every 8 connection(s)
I20260929 17:52:14.028828 25866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:14.031733 25934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:14.031818 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25866
I20260929 17:52:14.031950 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:14.032267 3807 external_mini_cluster.cc:1428] Setting key 9c608cd4b520aa76389df45d964c0f28
I20260929 17:52:14.038746 25934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3: Bootstrap starting.
I20260929 17:52:14.041329 25934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:14.041738 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:14.033677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55959 (local address 127.3.183.254:37795)
0929 17:52:14.034091 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:14.034107 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:14.034144 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:52:14.034916 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:14.034935 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:14.035294 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:14.035604 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:14.036044 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:14.037073 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:14.038274 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:14.038757 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:14.038893 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:14.040001 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:14.040006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:14.040019 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:14.040419 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:14.040478 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:14.040502 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:14.040577 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:14.040728 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 17:52:14.040926 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":129,"threads_started":1}
I20260929 17:52:14.042114 25934 log.cc:824] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:14.044147 25934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3: No bootstrap required, opened a new log
I20260929 17:52:14.044528 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:14.047500 25934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdeeeb105854f2cbfdef74f610119a3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37795 } }
I20260929 17:52:14.047735 25934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:14.047820 25934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdeeeb105854f2cbfdef74f610119a3, State: Initialized, Role: FOLLOWER
I20260929 17:52:14.048223 25934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdeeeb105854f2cbfdef74f610119a3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37795 } }
I20260929 17:52:14.048357 25934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:14.048422 25934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:14.048520 25934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:14.049419 25934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdeeeb105854f2cbfdef74f610119a3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37795 } }
I20260929 17:52:14.049788 25934 leader_election.cc:302] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bdeeeb105854f2cbfdef74f610119a3; no voters:
I20260929 17:52:14.050118 25934 leader_election.cc:288] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:14.050207 25944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:14.050421 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:14.046795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40501 (local address 127.3.183.254:37795)
0929 17:52:14.046920 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:14.046924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:14.046934 (+ 10us) server_negotiation.cc:407] Connection header received
0929 17:52:14.046965 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:14.046967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:14.047001 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:14.047069 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:14.047319 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:14.047933 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:14.048793 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:14.049041 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:14.049128 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:14.049970 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:14.049973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:14.049976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:14.049989 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:14.050015 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:14.050026 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:14.050087 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:14.050214 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 17:52:14.050267 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:52:14.050508 25944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [term 1 LEADER]: Becoming Leader. State: Replica: 1bdeeeb105854f2cbfdef74f610119a3, State: Running, Role: LEADER
I20260929 17:52:14.051229 25944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdeeeb105854f2cbfdef74f610119a3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37795 } }
I20260929 17:52:14.051668 25934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:14.053220 25945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bdeeeb105854f2cbfdef74f610119a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdeeeb105854f2cbfdef74f610119a3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37795 } } }
I20260929 17:52:14.053364 25945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:14.053676 25946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdeeeb105854f2cbfdef74f610119a3. Latest consensus state: current_term: 1 leader_uuid: "1bdeeeb105854f2cbfdef74f610119a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdeeeb105854f2cbfdef74f610119a3" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 37795 } } }
I20260929 17:52:14.053768 25946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3 [sys.catalog]: This master's current role is: LEADER
W20260929 17:52:14.056739 25959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:14.056994 25959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:52:14.057080 25956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:14.057794 25956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:14.063243 25956 catalog_manager.cc:1347] Generated new cluster ID: d1b11fd775fb4d01a8641c053c207d1b
I20260929 17:52:14.063321 25956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:14.081254 25956 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:14.082592 25956 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:14.087245 25956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bdeeeb105854f2cbfdef74f610119a3: Generated new TSK 0
I20260929 17:52:14.088057 25956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:14.094359 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25866
2026-09-29T17:52:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:14 dist-test-slave-wbpt krb5kdc[25965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:14 dist-test-slave-wbpt krb5kdc[25965](info): set up 2 sockets
Sep 29 17:52:14 dist-test-slave-wbpt krb5kdc[25965](info): commencing operation
krb5kdc: starting...
W20260929 17:52:15.059957 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:15 dist-test-slave-wbpt krb5kdc[25965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:15.180387 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34335
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:15.292582 25978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:15.292868 25978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:15.292912 25978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:15.297135 25978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:15.297210 25978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:15.297230 25978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:15.297247 25978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:15.297261 25978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:15.302914 25978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:34335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:15.304100 25978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:15.305125 25978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:15.313961 25984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:15.314077 25978 server_base.cc:1034] running on GCE node
W20260929 17:52:15.313961 25983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:15.314075 25986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:15.314939 25978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:15.316197 25978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:15.317401 25978 hybrid_clock.cc:648] HybridClock initialized: now 1790704335317381 us; error 34 us; skew 500 ppm
Sep 29 17:52:15 dist-test-slave-wbpt krb5kdc[25965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:15.324549 25978 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:15.325636 25978 webserver.cc:460] Webserver started at http://127.3.183.254:37957/ using document root <none> and password file <none>
I20260929 17:52:15.326229 25978 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:15.326306 25978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:15.326498 25978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:15.328856 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "af0feddc96514592a4369b74338c7ccc"
format_stamp: "Formatted at 2026-09-29 17:52:15 on dist-test-slave-wbpt"
server_key: "4f00501324657f1e14664f383487871c"
server_key_iv: "415b33512ed6025f627a8a435ad6de87"
server_key_version: "encryptionkey@0"
I20260929 17:52:15.329458 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "af0feddc96514592a4369b74338c7ccc"
format_stamp: "Formatted at 2026-09-29 17:52:15 on dist-test-slave-wbpt"
server_key: "4f00501324657f1e14664f383487871c"
server_key_iv: "415b33512ed6025f627a8a435ad6de87"
server_key_version: "encryptionkey@0"
I20260929 17:52:15.333662 25978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:52:15.336442 25993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:15.337893 25978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:52:15.338020 25978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "af0feddc96514592a4369b74338c7ccc"
format_stamp: "Formatted at 2026-09-29 17:52:15 on dist-test-slave-wbpt"
server_key: "4f00501324657f1e14664f383487871c"
server_key_iv: "415b33512ed6025f627a8a435ad6de87"
server_key_version: "encryptionkey@0"
I20260929 17:52:15.338138 25978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:15.343047 25978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:15.346553 25978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:15.346717 25978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:15.353747 25978 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:34335
I20260929 17:52:15.353767 26045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:34335 every 8 connection(s)
I20260929 17:52:15.354776 25978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:15.356738 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 25978
I20260929 17:52:15.356881 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:15.357173 3807 external_mini_cluster.cc:1428] Setting key 652a7a390e4f55343e4c65121eadad36
I20260929 17:52:15.357920 26046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:15.362681 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:15.358654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54569 (local address 127.3.183.254:34335)
0929 17:52:15.359161 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:15.359176 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:15.359213 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:52:15.359945 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:15.359972 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:15.360276 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:15.360571 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:15.361402 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:15.361406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:15.361419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:15.361430 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:15.361495 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:15.361522 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:15.361669 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:15.361776 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 17:52:15.361927 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":168,"threads_started":1}
I20260929 17:52:15.364477 26046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc: Bootstrap starting.
I20260929 17:52:15.365520 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:15.366791 26046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:15.367550 26046 log.cc:824] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:15.369120 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:15.367461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51083 (local address 127.3.183.254:34335)
0929 17:52:15.367591 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:15.367595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:15.367838 (+ 243us) server_negotiation.cc:407] Connection header received
0929 17:52:15.368009 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:15.368012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:15.368055 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:15.368145 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:15.368781 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:15.368784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:15.368786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:52:15.368796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:15.368820 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:52:15.368830 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:15.368893 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:15.368976 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 17:52:15.369007 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:52:15.369987 26046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc: No bootstrap required, opened a new log
I20260929 17:52:15.373796 26046 raft_consensus.cc:348] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0feddc96514592a4369b74338c7ccc" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34335 } }
I20260929 17:52:15.374066 26046 raft_consensus.cc:374] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:15.374104 26046 raft_consensus.cc:729] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0feddc96514592a4369b74338c7ccc, State: Initialized, Role: FOLLOWER
I20260929 17:52:15.374543 26046 consensus_queue.cc:260] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0feddc96514592a4369b74338c7ccc" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34335 } }
I20260929 17:52:15.374671 26046 raft_consensus.cc:388] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:15.374719 26046 raft_consensus.cc:482] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:15.374794 26046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:15.375932 26046 raft_consensus.cc:504] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0feddc96514592a4369b74338c7ccc" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34335 } }
I20260929 17:52:15.376298 26046 leader_election.cc:302] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0feddc96514592a4369b74338c7ccc; no voters:
I20260929 17:52:15.376672 26046 leader_election.cc:288] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:15.376863 26056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:15.377144 26056 raft_consensus.cc:686] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [term 1 LEADER]: Becoming Leader. State: Replica: af0feddc96514592a4369b74338c7ccc, State: Running, Role: LEADER
I20260929 17:52:15.377509 26056 consensus_queue.cc:237] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0feddc96514592a4369b74338c7ccc" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34335 } }
I20260929 17:52:15.377909 26046 sys_catalog.cc:564] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:15.379230 26057 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af0feddc96514592a4369b74338c7ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0feddc96514592a4369b74338c7ccc" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34335 } } }
I20260929 17:52:15.379290 26058 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader af0feddc96514592a4369b74338c7ccc. Latest consensus state: current_term: 1 leader_uuid: "af0feddc96514592a4369b74338c7ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0feddc96514592a4369b74338c7ccc" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 34335 } } }
I20260929 17:52:15.379379 26057 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:15.379402 26058 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:15.379842 26067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:15.383065 26067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:15.389384 26067 catalog_manager.cc:1347] Generated new cluster ID: f06fb5ade7644c1f8d301384310d9f77
I20260929 17:52:15.389492 26067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:15.422395 26067 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:15.422767 26067 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:15.428299 26067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af0feddc96514592a4369b74338c7ccc: Generated new TSK 0
I20260929 17:52:15.429121 26067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:52:15.440240 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 25978
2026-09-29T17:52:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1345 ms)
[----------] 18 tests from AuthTokenIssuingTest (23999 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:15 dist-test-slave-wbpt krb5kdc[26077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:15 dist-test-slave-wbpt krb5kdc[26077](info): set up 2 sockets
Sep 29 17:52:15 dist-test-slave-wbpt krb5kdc[26077](info): commencing operation
krb5kdc: starting...
W20260929 17:52:16.395991 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:16 dist-test-slave-wbpt krb5kdc[26077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:16.515550 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38951
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:38951,127.3.183.253:45501,127.3.183.252:40741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:16.625980 26090 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:52:16.626297 26090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:16.626320 26090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:16.626366 26090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:16.630506 26090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:16.630594 26090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:16.630612 26090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:16.630628 26090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:16.630641 26090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:16.636260 26090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:38951,127.3.183.253:45501,127.3.183.252:40741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:16.637374 26090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:16.638389 26090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:16.646977 26098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:16.647049 26095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:16.647003 26096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:16.647092 26090 server_base.cc:1034] running on GCE node
I20260929 17:52:16.647936 26090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:16.649082 26090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:16.650238 26090 hybrid_clock.cc:648] HybridClock initialized: now 1790704336650218 us; error 38 us; skew 500 ppm
Sep 29 17:52:16 dist-test-slave-wbpt krb5kdc[26077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:16.653038 26090 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:16.653995 26090 webserver.cc:460] Webserver started at http://127.3.183.254:35701/ using document root <none> and password file <none>
I20260929 17:52:16.654600 26090 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:16.654670 26090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:16.654863 26090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:16.657401 26090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "169c2ca596b147448ceba5a835fa8b9e"
format_stamp: "Formatted at 2026-09-29 17:52:16 on dist-test-slave-wbpt"
server_key: "f02e2984f09632864fd8fd18c97c4213"
server_key_iv: "da08c420fc435055548ea4c2b3af3b3a"
server_key_version: "encryptionkey@0"
I20260929 17:52:16.658051 26090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "169c2ca596b147448ceba5a835fa8b9e"
format_stamp: "Formatted at 2026-09-29 17:52:16 on dist-test-slave-wbpt"
server_key: "f02e2984f09632864fd8fd18c97c4213"
server_key_iv: "da08c420fc435055548ea4c2b3af3b3a"
server_key_version: "encryptionkey@0"
I20260929 17:52:16.662182 26090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:52:16.664808 26105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:16.666162 26090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:52:16.666288 26090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "169c2ca596b147448ceba5a835fa8b9e"
format_stamp: "Formatted at 2026-09-29 17:52:16 on dist-test-slave-wbpt"
server_key: "f02e2984f09632864fd8fd18c97c4213"
server_key_iv: "da08c420fc435055548ea4c2b3af3b3a"
server_key_version: "encryptionkey@0"
I20260929 17:52:16.666394 26090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:16.682160 26090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:16.685298 26090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:16.685484 26090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:16.692454 26090 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:38951
I20260929 17:52:16.692735 26157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:38951 every 8 connection(s)
I20260929 17:52:16.693427 26090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:16.696286 26158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:16.700202 26158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:16.702163 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 26090
I20260929 17:52:16.702311 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:16.702577 3807 external_mini_cluster.cc:1428] Setting key da0403aedabc18ac65f2d732e3566839
WARNING: no policy specified for kudu/127.3.183.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.253@KRBTEST.COM" created.
Sep 29 17:52:16 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:16.719120 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.702697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.3.183.254:38951)
0929 17:52:16.703312 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:16.703331 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:16.703491 (+ 160us) server_negotiation.cc:407] Connection header received
0929 17:52:16.704771 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:16.704777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:16.705048 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:16.705256 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:16.706172 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.707620 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.708766 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.709352 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.709540 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.714389 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:16.714395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:16.714407 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:16.714430 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:16.715996 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.716467 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.716472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.716474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.716578 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.716858 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.716860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.716862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.717281 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:16.717403 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:16.717765 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 17:52:16.717996 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"spinlock_wait_cycles":96128,"thread_start_us":177,"threads_started":1}
I20260929 17:52:16.719120 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.702697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38951 (local address 127.0.0.1:44688)
0929 17:52:16.703305 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.703373 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:16.704431 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:16.705337 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:16.705342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:16.705725 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:16.706012 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.706027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.707818 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.707822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.708632 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.708643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.709539 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.709542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.709644 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.710221 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:16.710238 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:16.714226 (+ 3988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:16.716099 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.716108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.716119 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.716351 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.716658 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.716660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.716663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.716770 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.717393 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:16.717399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:16.717531 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 17:52:16.717674 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 17:52:16.718082 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":89,"thread_start_us":176,"threads_started":1}
I20260929 17:52:16.721730 26158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:16.722680 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.722294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35860)
0929 17:52:16.722489 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.722587 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 17:52:16.722931 26107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
I20260929 17:52:16.723850 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.723607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35874)
0929 17:52:16.723708 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.723755 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 17:52:16.724001 26158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:45501: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab.
Entry for principal kudu/127.3.183.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab.
I20260929 17:52:16.731097 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:45501
--webserver_interface=127.3.183.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab
--principal=kudu/127.3.183.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:38951,127.3.183.253:45501,127.3.183.252:40741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:52:16.773247 26158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } attempt: 1
I20260929 17:52:16.774153 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.773816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35886)
0929 17:52:16.773996 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.774070 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 17:52:16.774941 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.774710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35888)
0929 17:52:16.774817 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.774876 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 17:52:16.775064 26158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:45501: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
W20260929 17:52:16.841578 26165 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:52:16.841892 26165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:16.841929 26165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:16.842005 26165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 17:52:16.843238 26158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } attempt: 2
I20260929 17:52:16.844038 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.843721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35902)
0929 17:52:16.843906 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.843963 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 17:52:16.844767 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.844530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35916)
0929 17:52:16.844656 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.844707 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 17:52:16.844903 26158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:45501: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45501: connect: Connection refused (error 111)
W20260929 17:52:16.846468 26165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:16.846539 26165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:16.846561 26165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:16.846578 26165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:16.846592 26165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:16.852348 26165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:38951,127.3.183.253:45501,127.3.183.252:40741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab
--principal=kudu/127.3.183.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.183.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:16.853406 26165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:16.854393 26165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:52:16.863253 26165 server_base.cc:1034] running on GCE node
W20260929 17:52:16.863287 26170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:16.863304 26171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:16.863304 26173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:16.864096 26165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:16.865213 26165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:16.866382 26165 hybrid_clock.cc:648] HybridClock initialized: now 1790704336866355 us; error 38 us; skew 500 ppm
Sep 29 17:52:16 dist-test-slave-wbpt krb5kdc[26077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:16.869150 26165 init.cc:377] Logged in from keytab as kudu/127.3.183.253@KRBTEST.COM (short username kudu)
I20260929 17:52:16.870132 26165 webserver.cc:460] Webserver started at http://127.3.183.253:36455/ using document root <none> and password file <none>
I20260929 17:52:16.870672 26165 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:16.870736 26165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:16.870931 26165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:16.873042 26165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data/instance:
uuid: "180281a9160d48ccbb3eb06bd66db26b"
format_stamp: "Formatted at 2026-09-29 17:52:16 on dist-test-slave-wbpt"
server_key: "efe3679f0481069fc71ce2fb721bca13"
server_key_iv: "6b977ddee76467d064c044b65aed3edc"
server_key_version: "encryptionkey@0"
I20260929 17:52:16.873584 26165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/wal/instance:
uuid: "180281a9160d48ccbb3eb06bd66db26b"
format_stamp: "Formatted at 2026-09-29 17:52:16 on dist-test-slave-wbpt"
server_key: "efe3679f0481069fc71ce2fb721bca13"
server_key_iv: "6b977ddee76467d064c044b65aed3edc"
server_key_version: "encryptionkey@0"
I20260929 17:52:16.877385 26165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:52:16.879864 26180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:16.881112 26165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:16.881237 26165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/wal
uuid: "180281a9160d48ccbb3eb06bd66db26b"
format_stamp: "Formatted at 2026-09-29 17:52:16 on dist-test-slave-wbpt"
server_key: "efe3679f0481069fc71ce2fb721bca13"
server_key_iv: "6b977ddee76467d064c044b65aed3edc"
server_key_version: "encryptionkey@0"
I20260929 17:52:16.881342 26165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:16.898260 26165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:16.901454 26165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:16.901652 26165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:16.908779 26165 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.253:45501
I20260929 17:52:16.909070 26232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.253:45501 every 8 connection(s)
I20260929 17:52:16.909857 26165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
I20260929 17:52:16.912683 26233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:16.916538 26233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:16.917750 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 26165
I20260929 17:52:16.917861 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-1/wal/instance
I20260929 17:52:16.918162 3807 external_mini_cluster.cc:1428] Setting key c5c94db52eab2cb5ed36c8d15831e039
WARNING: no policy specified for kudu/127.3.183.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.252@KRBTEST.COM" created.
Sep 29 17:52:16 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:16.932577 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.918456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.3.183.254:38951)
0929 17:52:16.918582 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:16.918586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:16.919727 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 17:52:16.921168 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:16.921172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:16.921232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:16.921304 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:16.922566 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.923089 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.924268 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.924509 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.924601 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.929091 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:16.929095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:16.929097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:16.929125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:16.930419 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.930924 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.930928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.930929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.931008 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.931335 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.931339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.931340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.931566 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:16.931653 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:16.932261 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 17:52:16.932410 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:52:16.933393 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.919010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38951 (local address 127.0.0.1:44690)
0929 17:52:16.919552 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.919599 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:16.920911 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:16.921506 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:16.921513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:16.921871 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:16.922418 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.922432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.923209 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.923212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.924150 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.924160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.924657 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.924661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.924828 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.925442 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:16.925468 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:16.928851 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:16.930528 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.930534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.930547 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.930799 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.931089 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.931092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.931094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.931236 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.931669 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:16.931676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:16.932003 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 17:52:16.932170 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 17:52:16.932512 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":151,"threads_started":1}
I20260929 17:52:16.935178 26233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:16.936062 26158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } attempt: 3
Sep 29 17:52:16 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
Sep 29 17:52:16 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab.
Entry for principal kudu/127.3.183.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab.
I20260929 17:52:16.943187 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:40741
--webserver_interface=127.3.183.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab
--principal=kudu/127.3.183.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:38951,127.3.183.253:45501,127.3.183.252:40741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:52:16.945724 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.936428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35932)
0929 17:52:16.936526 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.936539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:16.936611 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:16.937035 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:16.937037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:16.937197 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:16.937300 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.937308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.938268 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.938272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.938852 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.938891 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.939553 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.939557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.939666 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.940378 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:16.940393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:16.941953 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:16.944135 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.944141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.944330 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.944614 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.944619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.944714 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.945243 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:16.945246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:16.945349 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 17:52:16.945425 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:52:16.945560 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 17:52:16.945714 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.935730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35920)
0929 17:52:16.935913 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.935926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:16.935998 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:16.936730 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:16.936733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:16.936905 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:16.937051 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.937058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.938315 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.938318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.938838 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:16.938846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.939591 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.939594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:16.939708 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.940212 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:16.940223 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:16.941521 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:16.944219 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.944224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.944427 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.944675 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:16.944680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:16.944749 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:16.945219 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:16.945222 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:16.945309 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 17:52:16.945396 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:52:16.945543 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":17}
I20260929 17:52:16.945832 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.935904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.3.183.253:45501)
0929 17:52:16.936282 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:16.936293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:16.936312 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:52:16.936360 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:16.936397 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:16.936610 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:16.936751 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:16.937153 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.938228 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.938969 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.939464 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.939580 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.941718 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:16.941729 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:16.941746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:16.941770 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:16.944118 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944517 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.944521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.944522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.944597 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944824 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.944826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.944827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.945122 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:16.945240 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:16.945526 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 17:52:16.945660 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":196,"server-negotiator.queue_time_us":240,"spinlock_wait_cycles":6528,"thread_start_us":100,"threads_started":1}
I20260929 17:52:16.946578 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.936496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.3.183.253:45501)
0929 17:52:16.936823 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:16.936831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:16.936842 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:52:16.936869 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:16.936872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:16.936910 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:16.936992 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:16.937413 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.938159 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.938986 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:16.939426 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:16.939528 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:16.942136 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:16.942140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:16.942143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:16.942165 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:16.944037 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944429 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.944435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.944437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.944509 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:16.944816 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:16.944818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:16.944820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:16.945135 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:16.946259 (+ 1124us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:16.946333 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 17:52:16.946455 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":268,"thread_start_us":141,"threads_started":1}
I20260929 17:52:16.947680 26233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:16.947692 26158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:16.948433 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.948172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38106)
0929 17:52:16.948305 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.948361 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 17:52:16.948480 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.948210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38114)
0929 17:52:16.948358 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.948423 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 17:52:16.948645 26184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
I20260929 17:52:16.949214 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.948923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38128)
0929 17:52:16.949101 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.949148 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260929 17:52:16.949375 26158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:40741: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
I20260929 17:52:16.949600 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.949226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38140)
0929 17:52:16.949505 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.949546 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
W20260929 17:52:16.949695 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:40741: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
I20260929 17:52:16.980541 26158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } attempt: 1
I20260929 17:52:16.981420 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.981091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38146)
0929 17:52:16.981253 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.981326 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 17:52:16.982165 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.981944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38160)
0929 17:52:16.982048 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.982099 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 17:52:16.982292 26158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:40741: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
I20260929 17:52:16.998996 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } attempt: 1
I20260929 17:52:17.000000 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:16.999565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38162)
0929 17:52:16.999749 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:16.999884 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 17:52:17.000816 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.000576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38174)
0929 17:52:17.000696 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.000747 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 17:52:17.001003 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:40741: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
W20260929 17:52:17.056957 26240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:52:17.057310 26240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:17.057355 26240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:17.057417 26240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 17:52:17.057446 26158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } attempt: 2
I20260929 17:52:17.058308 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.057976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38188)
0929 17:52:17.058146 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.058216 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 17:52:17.058983 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.058782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38198)
0929 17:52:17.058872 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.058919 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 17:52:17.059149 26158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:40741: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
W20260929 17:52:17.062074 26240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:17.062143 26240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:17.062171 26240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:17.062202 26240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:17.062230 26240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:17.067832 26240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:38951,127.3.183.253:45501,127.3.183.252:40741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab
--principal=kudu/127.3.183.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:40741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.183.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:17.068972 26240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:17.069195 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } attempt: 2
I20260929 17:52:17.070053 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.069734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38200)
0929 17:52:17.069901 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.069975 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 17:52:17.070245 26240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:52:17.070763 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.070533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38204)
0929 17:52:17.070640 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.070687 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 17:52:17.070914 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:40741: Network error: Client connection negotiation failed: client connection to 127.3.183.252:40741: connect: Connection refused (error 111)
W20260929 17:52:17.078476 26248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:17.078473 26247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:17.078652 26240 server_base.cc:1034] running on GCE node
W20260929 17:52:17.078812 26250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:17.079319 26240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:17.080456 26240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:17.081617 26240 hybrid_clock.cc:648] HybridClock initialized: now 1790704337081599 us; error 30 us; skew 500 ppm
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:17.084252 26240 init.cc:377] Logged in from keytab as kudu/127.3.183.252@KRBTEST.COM (short username kudu)
I20260929 17:52:17.085142 26240 webserver.cc:460] Webserver started at http://127.3.183.252:40563/ using document root <none> and password file <none>
I20260929 17:52:17.085672 26240 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:17.085736 26240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:17.085919 26240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:17.087975 26240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data/instance:
uuid: "7b03914df3764f6db602fdff85c4a12b"
format_stamp: "Formatted at 2026-09-29 17:52:17 on dist-test-slave-wbpt"
server_key: "557db8bfca71eccef8da30fdee975b90"
server_key_iv: "d5b29c41340b083191b83ec8afdeb634"
server_key_version: "encryptionkey@0"
I20260929 17:52:17.088498 26240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/wal/instance:
uuid: "7b03914df3764f6db602fdff85c4a12b"
format_stamp: "Formatted at 2026-09-29 17:52:17 on dist-test-slave-wbpt"
server_key: "557db8bfca71eccef8da30fdee975b90"
server_key_iv: "d5b29c41340b083191b83ec8afdeb634"
server_key_version: "encryptionkey@0"
I20260929 17:52:17.092039 26240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:52:17.094313 26257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:17.095507 26240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:52:17.095618 26240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/wal
uuid: "7b03914df3764f6db602fdff85c4a12b"
format_stamp: "Formatted at 2026-09-29 17:52:17 on dist-test-slave-wbpt"
server_key: "557db8bfca71eccef8da30fdee975b90"
server_key_iv: "d5b29c41340b083191b83ec8afdeb634"
server_key_version: "encryptionkey@0"
I20260929 17:52:17.095717 26240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:17.121134 26240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:17.123839 26240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:17.124013 26240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:17.130533 26240 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.252:40741
I20260929 17:52:17.130781 26309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.252:40741 every 8 connection(s)
I20260929 17:52:17.131513 26240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
I20260929 17:52:17.134173 26310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:17.137869 26310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:17.141415 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 26240
I20260929 17:52:17.141525 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704058755242-3807-0/minicluster-data/master-2/wal/instance
I20260929 17:52:17.141758 3807 external_mini_cluster.cc:1428] Setting key 7f579295e05bc6e4d2f01ad7c4bd71ba
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:52:17.151640 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.139619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.3.183.254:38951)
0929 17:52:17.139741 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.139744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.140570 (+ 826us) server_negotiation.cc:407] Connection header received
0929 17:52:17.141533 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.141536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.141597 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.141671 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.142977 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.143484 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.144566 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.144775 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.144864 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.147505 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.147508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.147510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.147536 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.149557 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.150144 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.150146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.150148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.150223 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.150572 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.150574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.150575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.150736 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.150815 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.151405 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.151520 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:52:17.152825 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.139977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:38951 (local address 127.0.0.1:44692)
0929 17:52:17.140421 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.140465 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:17.141325 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:17.141868 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:17.141878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:17.142232 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:17.142825 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.142842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.143599 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.143602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.144454 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.144464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.144915 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.144920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.145120 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.145681 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:17.145701 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:17.147360 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:17.149660 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.149671 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.149683 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.150042 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.150311 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.150315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.150317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.150480 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.150853 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:17.150859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:17.151125 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 17:52:17.151332 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 17:52:17.151643 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260929 17:52:17.152830 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.142369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.3.183.252:40741)
0929 17:52:17.142688 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.142699 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.142723 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:52:17.142781 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.142786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.142948 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.143080 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.143758 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.144558 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.145482 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.145950 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.146046 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.148950 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.148958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.148972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.148999 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.150737 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.151108 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.151113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.151115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.151201 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.151465 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.151469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.151470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.151748 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.152002 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.152164 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.152319 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":147,"threads_started":1}
I20260929 17:52:17.154152 26310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:17.158324 26158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } attempt: 3
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:17.162087 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } attempt: 3
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:52:17.163600 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.154767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.3.183.253:45501)
0929 17:52:17.154889 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.154893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.154938 (+ 45us) server_negotiation.cc:407] Connection header received
0929 17:52:17.155071 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.155075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.155118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.155178 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.155593 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.156253 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.156982 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.157233 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.157321 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.159590 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.159594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.159597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.159624 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.161850 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.162317 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.162321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.162323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.162417 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.162675 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.162679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.162680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.162879 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.162970 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.163298 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.163423 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:52:17.164053 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.154642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45501 (local address 127.0.0.1:35944)
0929 17:52:17.154884 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.154898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:17.154975 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:17.155202 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:17.155205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:17.155369 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:17.155480 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.155486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.156353 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.156357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.156880 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.156887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.157330 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.157333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.157430 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.157868 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:17.157881 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:17.159457 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:17.161946 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.161950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.161952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.162219 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.162515 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.162518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.162520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.162605 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.163020 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:17.163023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:17.163118 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 17:52:17.163738 (+ 620us) client_negotiation.cc:240] Negotiation successful
0929 17:52:17.163902 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260929 17:52:17.164768 26310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:17.166426 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.155369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.3.183.254:38951)
0929 17:52:17.155493 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.155498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.155655 (+ 157us) server_negotiation.cc:407] Connection header received
0929 17:52:17.155741 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.155744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.155809 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.155884 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.156461 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.156976 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.157745 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.157986 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.158080 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.160769 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.160779 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.160781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.160805 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.163471 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.164683 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.164687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.164689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.164782 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.165111 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.165115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.165117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.165333 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.165573 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.165907 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.166027 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:52:17.167114 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.158772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38224)
0929 17:52:17.159036 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.159047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:17.159130 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:17.159362 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:17.159366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:17.159545 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:17.159657 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.159666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.160621 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.160624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.161186 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.161194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.161750 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.161754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.161852 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.162212 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:17.162228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:17.163698 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:17.165497 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.165500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.165502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.165720 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.166031 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.166034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.166037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.166162 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.166680 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:17.166683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:17.166775 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 17:52:17.166855 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:52:17.166976 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 17:52:17.167721 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.158838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.3.183.252:40741)
0929 17:52:17.158949 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.158953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.159105 (+ 152us) server_negotiation.cc:407] Connection header received
0929 17:52:17.159197 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.159200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.159239 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.159307 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.159806 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.160455 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.161305 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.161644 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.161750 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.163920 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.163924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.163926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.163949 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.165398 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.165835 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.165837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.165839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.165917 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.166270 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.166273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.166275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.166573 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.167382 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.167472 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.167578 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:52:17.171446 26158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e: Bootstrap starting.
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:52:17.174419 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.162527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38234)
0929 17:52:17.162907 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.162923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:17.163011 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:17.164088 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:17.164091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:17.164284 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:17.164448 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.164456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.166679 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.166683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.167555 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.167567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.168233 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.168237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.168363 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.168839 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:17.168855 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:17.170476 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:17.172595 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.172599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.172601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.172858 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.173211 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.173215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.173217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.173327 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.173858 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:17.173861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:17.173978 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 17:52:17.174076 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 17:52:17.174221 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
Sep 29 17:52:17 dist-test-slave-wbpt krb5kdc[26077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
I20260929 17:52:17.174630 26158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:17.175151 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.162629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.3.183.252:40741)
0929 17:52:17.163831 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.163842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.163856 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:52:17.163891 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.163894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.163938 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.164571 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.164674 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.166549 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.167691 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.168085 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.168596 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.170639 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.170643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.170646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.170670 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.172464 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.172997 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.173000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.173002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.173101 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.173443 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.173446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.173448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.173732 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.174727 (+ 995us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.174812 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.174941 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":177,"threads_started":1}
I20260929 17:52:17.175709 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.165264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:40741 (local address 127.0.0.1:38246)
0929 17:52:17.166714 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:17.166729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:17.166810 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:17.167169 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:17.167172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:17.167341 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:17.167425 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.167431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.168432 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.168435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.169415 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:17.169424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.170040 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.170043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:17.170114 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.170509 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:17.170522 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:17.172431 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:17.174391 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.174394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.174396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.174604 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.174844 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:17.174847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:17.174848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:17.174911 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:17.175307 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:17.175309 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:17.175381 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 17:52:17.175457 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:52:17.175568 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1384}
I20260929 17:52:17.176106 26158 log.cc:824] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:17.176404 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.165324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.3.183.252:40741)
0929 17:52:17.165585 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.165595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.166892 (+ 1297us) server_negotiation.cc:407] Connection header received
0929 17:52:17.166922 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.166925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.166964 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.167040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.167543 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.168331 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.169529 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.169849 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.169935 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.172541 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.172544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.172547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.172567 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.174287 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.174699 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.174702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.174704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.174775 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.175001 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.175003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.175005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.175192 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.175278 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.175900 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.176237 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260929 17:52:17.178407 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: Bootstrap starting.
I20260929 17:52:17.178514 26158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e: No bootstrap required, opened a new log
I20260929 17:52:17.179147 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.168130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.3.183.253:45501)
0929 17:52:17.168747 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.168752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.168903 (+ 151us) server_negotiation.cc:407] Connection header received
0929 17:52:17.169059 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.169062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.169103 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.169147 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.169988 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.170794 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.171933 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.172238 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.172312 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.175702 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.175705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.175708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.175731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.177154 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.177655 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.177658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.177660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.177748 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.178100 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.178104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.178106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.178334 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.178479 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.178878 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.178997 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541}
I20260929 17:52:17.180028 26310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: Bootstrap starting.
I20260929 17:52:17.181789 26233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:17.182366 26310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:17.182574 26158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
I20260929 17:52:17.182704 26233 log.cc:824] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:17.182793 26158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169c2ca596b147448ceba5a835fa8b9e, State: Initialized, Role: FOLLOWER
I20260929 17:52:17.183049 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:17.183151 26310 log.cc:824] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:17.183117 26158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
I20260929 17:52:17.183627 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.183809 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:17.184326 26132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "169c2ca596b147448ceba5a835fa8b9e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44696
I20260929 17:52:17.184517 26158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:17.184640 26132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 17:52:17.184769 26132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:17.186004 26310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: No bootstrap required, opened a new log
I20260929 17:52:17.186000 26132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
I20260929 17:52:17.188442 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: No bootstrap required, opened a new log
I20260929 17:52:17.190052 26310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
I20260929 17:52:17.190284 26310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b03914df3764f6db602fdff85c4a12b, State: Initialized, Role: FOLLOWER
I20260929 17:52:17.190610 26310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
I20260929 17:52:17.191090 26333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.191238 26333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:17.191919 26310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:17.192760 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "169c2ca596b147448ceba5a835fa8b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b03914df3764f6db602fdff85c4a12b"
I20260929 17:52:17.193009 26284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:17.192903 26233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
I20260929 17:52:17.193122 26233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180281a9160d48ccbb3eb06bd66db26b, State: Initialized, Role: FOLLOWER
I20260929 17:52:17.193455 26233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
I20260929 17:52:17.194391 26284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169c2ca596b147448ceba5a835fa8b9e in term 1.
I20260929 17:52:17.194720 26233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:17.194829 26206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "169c2ca596b147448ceba5a835fa8b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "180281a9160d48ccbb3eb06bd66db26b"
I20260929 17:52:17.195031 26206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:17.186957 26132 leader_election.cc:288] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [CANDIDATE]: Term 1 election: Requested vote from peers 180281a9160d48ccbb3eb06bd66db26b (127.3.183.253:45501), 7b03914df3764f6db602fdff85c4a12b (127.3.183.252:40741)
W20260929 17:52:17.187011 26111 tablet.cc:2369] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:52:17.190100 26331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:17.198238 26331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:52:17.200712 26206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169c2ca596b147448ceba5a835fa8b9e in term 1.
I20260929 17:52:17.203392 26109 leader_election.cc:302] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169c2ca596b147448ceba5a835fa8b9e, 7b03914df3764f6db602fdff85c4a12b; no voters:
I20260929 17:52:17.203758 26319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:17.204051 26319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [term 1 LEADER]: Becoming Leader. State: Replica: 169c2ca596b147448ceba5a835fa8b9e, State: Running, Role: LEADER
W20260929 17:52:17.204444 26356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 17:52:17.204428 26319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } }
W20260929 17:52:17.204614 26356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:52:17.206324 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169c2ca596b147448ceba5a835fa8b9e. Latest consensus state: current_term: 1 leader_uuid: "169c2ca596b147448ceba5a835fa8b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.206444 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:17.208523 26090 master_runner.cc:263] No leader master found from master 169c2ca596b147448ceba5a835fa8b9e
I20260929 17:52:17.208581 26090 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:52:17.201797 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.208747 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [sys.catalog]: This master's current role is: FOLLOWER
W20260929 17:52:17.209824 26345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:17.209916 26345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:52:17.211030 26240 master_runner.cc:263] No leader master found from master 7b03914df3764f6db602fdff85c4a12b
I20260929 17:52:17.211095 26240 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:52:17.211911 26365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:17.212695 26365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:17.217937 26284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [term 1 FOLLOWER]: Refusing update from remote peer 169c2ca596b147448ceba5a835fa8b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:52:17.218820 26206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [term 1 FOLLOWER]: Refusing update from remote peer 169c2ca596b147448ceba5a835fa8b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:52:17.219529 26319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:52:17.220402 26319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:52:17.224426 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.206370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.3.183.252:40741)
0929 17:52:17.206485 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.206490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.214661 (+ 8171us) server_negotiation.cc:407] Connection header received
0929 17:52:17.214801 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.214804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.214856 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.214934 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.216047 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.216859 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.218033 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.218402 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.218510 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.220511 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.220516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.220519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.220548 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.222288 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.222829 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.222833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.222835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.222932 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.223256 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.223259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.223261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.223494 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.223637 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.224118 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.224248 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:52:17.227080 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169c2ca596b147448ceba5a835fa8b9e. Latest consensus state: current_term: 1 leader_uuid: "169c2ca596b147448ceba5a835fa8b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.227224 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:17.227185 26333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169c2ca596b147448ceba5a835fa8b9e. Latest consensus state: current_term: 1 leader_uuid: "169c2ca596b147448ceba5a835fa8b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.227499 26333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:17.232092 26319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169c2ca596b147448ceba5a835fa8b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.232210 26319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:17.233691 26333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169c2ca596b147448ceba5a835fa8b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.233804 26333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:17.233990 26369 mvcc.cc:205] Tried to move back new op lower bound from 7334724965236224000 to 7334724965193125888. Current Snapshot: MvccSnapshot[applied={T|T < 7334724965236224000}]
I20260929 17:52:17.235288 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.207880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.3.183.253:45501)
0929 17:52:17.207996 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.208001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.226093 (+ 18092us) server_negotiation.cc:407] Connection header received
0929 17:52:17.226286 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.226290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.226344 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.226401 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.227303 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.228081 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.229164 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.229471 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.229553 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.231208 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.231212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.231215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.231243 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.233346 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.233860 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.233864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.233866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.233956 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.234264 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.234267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.234269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.234496 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.234597 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.234992 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.235118 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:52:17.235392 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:17.205859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.3.183.254:38951)
0929 17:52:17.205977 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.205982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.214997 (+ 9015us) server_negotiation.cc:407] Connection header received
0929 17:52:17.215181 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.215185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.215232 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.215300 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.216221 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.216941 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.224869 (+ 7928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:17.225198 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:17.225295 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:17.231931 (+ 6636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:17.231936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:17.231939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:17.231972 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:17.233565 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.234013 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.234017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.234019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.234107 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:17.234427 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:17.234431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:17.234433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:17.234644 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:17.234790 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:17.235124 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 17:52:17.235237 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:52:17.237489 26370 mvcc.cc:205] Tried to move back new op lower bound from 7334724965236224000 to 7334724965193125888. Current Snapshot: MvccSnapshot[applied={T|T < 7334724965236224000}]
I20260929 17:52:17.237560 26365 catalog_manager.cc:1347] Generated new cluster ID: 9e3e3f527f764b63b524dec2c66ec634
I20260929 17:52:17.237609 26365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:17.252247 26363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169c2ca596b147448ceba5a835fa8b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.252430 26363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:17.253891 26372 mvcc.cc:205] Tried to move back new op lower bound from 7334724965236224000 to 7334724965193125888. Current Snapshot: MvccSnapshot[applied={T|T < 7334724965236224000}]
I20260929 17:52:17.254261 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169c2ca596b147448ceba5a835fa8b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c2ca596b147448ceba5a835fa8b9e" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 38951 } } peers { permanent_uuid: "180281a9160d48ccbb3eb06bd66db26b" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45501 } } peers { permanent_uuid: "7b03914df3764f6db602fdff85c4a12b" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 40741 } } }
I20260929 17:52:17.254380 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:17.285828 26365 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:17.287173 26365 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:17.307901 26365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 169c2ca596b147448ceba5a835fa8b9e: Generated new TSK 0
I20260929 17:52:17.310973 26365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:52:17.345270 26316 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.342025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.3.183.252:40741)
0929 17:52:17.342181 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.342187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.344099 (+ 1912us) server_negotiation.cc:407] Connection header received
0929 17:52:17.344324 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.344327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.344379 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.344434 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.345125 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38270: BlockingRecv error: recv got EOF from 127.0.0.1:38270 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:52:17.346658 26161 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.342807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.3.183.254:38951)
0929 17:52:17.342926 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.342930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.345748 (+ 2818us) server_negotiation.cc:407] Connection header received
0929 17:52:17.345927 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.345931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.345995 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.346047 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.346537 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44722: BlockingRecv error: recv got EOF from 127.0.0.1:44722 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 17:52:17.348165 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.343444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.3.183.253:45501)
0929 17:52:17.343564 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.343568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.347064 (+ 3496us) server_negotiation.cc:407] Connection header received
0929 17:52:17.347393 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.347397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.347451 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.347505 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.348047 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35982: BlockingRecv error: recv got EOF from 127.0.0.1:35982 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 17:52:17.352639 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.351127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.3.183.253:45501)
0929 17:52:17.351246 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.351250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.351691 (+ 441us) server_negotiation.cc:407] Connection header received
0929 17:52:17.351933 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.351936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.352026 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.352078 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.352546 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35992: BlockingRecv error: recv got EOF from 127.0.0.1:35992 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 17:52:17.357396 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.356054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.3.183.253:45501)
0929 17:52:17.356149 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.356153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.356584 (+ 431us) server_negotiation.cc:407] Connection header received
0929 17:52:17.356738 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.356741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.356787 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.356832 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.357303 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36006: BlockingRecv error: recv got EOF from 127.0.0.1:36006 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 17:52:17.362823 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.361386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.3.183.253:45501)
0929 17:52:17.361486 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.361490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.361995 (+ 505us) server_negotiation.cc:407] Connection header received
0929 17:52:17.362190 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.362193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.362241 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.362286 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.362726 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36018: BlockingRecv error: recv got EOF from 127.0.0.1:36018 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 17:52:17.370723 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.369059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.3.183.253:45501)
0929 17:52:17.369207 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.369212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.369692 (+ 480us) server_negotiation.cc:407] Connection header received
0929 17:52:17.369927 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.369930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.369996 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.370049 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.370610 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36030: BlockingRecv error: recv got EOF from 127.0.0.1:36030 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 17:52:17.382076 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.380931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.3.183.253:45501)
0929 17:52:17.381061 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.381066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.381295 (+ 229us) server_negotiation.cc:407] Connection header received
0929 17:52:17.381424 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.381428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.381479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.381543 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.381957 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36042: BlockingRecv error: recv got EOF from 127.0.0.1:36042 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 17:52:17.401330 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.400095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.3.183.253:45501)
0929 17:52:17.400217 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.400222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.400537 (+ 315us) server_negotiation.cc:407] Connection header received
0929 17:52:17.400676 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.400679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.400728 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.400789 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.401213 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36054: BlockingRecv error: recv got EOF from 127.0.0.1:36054 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 17:52:17.436492 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.435269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.3.183.253:45501)
0929 17:52:17.435394 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.435398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.435623 (+ 225us) server_negotiation.cc:407] Connection header received
0929 17:52:17.435802 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.435806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.435862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.435927 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.436364 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36062: BlockingRecv error: recv got EOF from 127.0.0.1:36062 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 17:52:17.503624 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.502546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.3.183.253:45501)
0929 17:52:17.502667 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.502670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.502846 (+ 176us) server_negotiation.cc:407] Connection header received
0929 17:52:17.502964 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.502967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.503017 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.503097 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.503517 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36064: BlockingRecv error: recv got EOF from 127.0.0.1:36064 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:52:17.634819 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.633721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.3.183.253:45501)
0929 17:52:17.633883 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.633888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.633979 (+ 91us) server_negotiation.cc:407] Connection header received
0929 17:52:17.634131 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.634134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.634182 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.634249 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.634716 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36070: BlockingRecv error: recv got EOF from 127.0.0.1:36070 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 17:52:17.894151 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:17.893061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.3.183.253:45501)
0929 17:52:17.893218 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:17.893223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:17.893255 (+ 32us) server_negotiation.cc:407] Connection header received
0929 17:52:17.893418 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:17.893420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:17.893470 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:17.893563 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:17.894053 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36074: BlockingRecv error: recv got EOF from 127.0.0.1:36074 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:52:18.205945 26356 catalog_manager.cc:1259] Loaded cluster ID: 9e3e3f527f764b63b524dec2c66ec634
I20260929 17:52:18.206039 26356 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: loading cluster ID for follower catalog manager: success
I20260929 17:52:18.207819 26356 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: acquiring CA information for follower catalog manager: success
I20260929 17:52:18.209045 26356 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 180281a9160d48ccbb3eb06bd66db26b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:52:18.211058 26345 catalog_manager.cc:1259] Loaded cluster ID: 9e3e3f527f764b63b524dec2c66ec634
I20260929 17:52:18.211125 26345 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: loading cluster ID for follower catalog manager: success
I20260929 17:52:18.213090 26345 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: acquiring CA information for follower catalog manager: success
I20260929 17:52:18.214319 26345 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b03914df3764f6db602fdff85c4a12b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:52:18.412065 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:18.408265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.3.183.253:45501)
0929 17:52:18.408507 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:18.408511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:18.408639 (+ 128us) server_negotiation.cc:407] Connection header received
0929 17:52:18.408727 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:18.408730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:18.408853 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:18.408961 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:18.409213 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.409746 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.410602 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.410828 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.410913 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:18.411188 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:18.411627 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:18.411718 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:18.411828 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 17:52:18.411883 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260929 17:52:18.419199 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:18.415309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.3.183.252:40741)
0929 17:52:18.415564 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:18.415567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:18.415581 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:52:18.415682 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:18.415685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:18.415842 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:18.415952 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:18.416165 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.416672 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.417602 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.417933 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.418034 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:18.418343 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:18.418813 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:18.418926 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:18.419000 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 17:52:18.419049 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260929 17:52:18.426609 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:18.422589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.3.183.253:45501)
0929 17:52:18.422723 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:18.422727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:18.423030 (+ 303us) server_negotiation.cc:407] Connection header received
0929 17:52:18.423210 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:18.423213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:18.423262 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:18.423319 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:18.423681 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.424407 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.425304 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.425642 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.425739 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:18.426153 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:18.426231 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:18.426304 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:18.426409 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 17:52:18.426469 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:52:18.426667 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:18.422265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.3.183.254:38951)
0929 17:52:18.422549 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:18.422553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:18.422617 (+ 64us) server_negotiation.cc:407] Connection header received
0929 17:52:18.422817 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:18.422820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:18.422989 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:18.423110 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:18.423322 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.424033 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.425359 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.425597 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.425684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:18.426001 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:18.426341 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:18.426429 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:18.426490 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 17:52:18.426546 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260929 17:52:18.426827 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:18.422286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.3.183.252:40741)
0929 17:52:18.422398 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:18.422403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:18.422947 (+ 544us) server_negotiation.cc:407] Connection header received
0929 17:52:18.423042 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:18.423045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:18.423092 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:18.423154 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:18.423437 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.424165 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.425362 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:18.425689 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:18.425768 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:18.426173 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:18.426242 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:18.426309 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:18.426659 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 17:52:18.426716 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:52:18.429867 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 26090
I20260929 17:52:18.435853 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 26165
I20260929 17:52:18.441147 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 26240
2026-09-29T17:52:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3002 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:18 dist-test-slave-wbpt krb5kdc[26459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:18 dist-test-slave-wbpt krb5kdc[26459](info): set up 2 sockets
Sep 29 17:52:18 dist-test-slave-wbpt krb5kdc[26459](info): commencing operation
krb5kdc: starting...
W20260929 17:52:19.403880 3807 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/test-user.keytab.
Sep 29 17:52:19 dist-test-slave-wbpt krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.183.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
Entry for principal kudu/127.3.183.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab.
I20260929 17:52:19.526269 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:36393
--webserver_interface=127.3.183.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:36393,127.3.183.253:45421,127.3.183.252:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:19.634469 26472 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:52:19.634790 26472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:19.634840 26472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:19.634905 26472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:19.639078 26472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:19.639158 26472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:19.639183 26472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:19.639204 26472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:19.639225 26472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:19.644714 26472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:36393,127.3.183.253:45421,127.3.183.252:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.254.keytab
--principal=kudu/127.3.183.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.254:36393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.183.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:19.645838 26472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:19.646888 26472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:19.655583 26478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:19.655583 26480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:19.655577 26477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:19.655972 26472 server_base.cc:1034] running on GCE node
I20260929 17:52:19.656566 26472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:19.657773 26472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:19.658965 26472 hybrid_clock.cc:648] HybridClock initialized: now 1790704339658945 us; error 33 us; skew 500 ppm
Sep 29 17:52:19 dist-test-slave-wbpt krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:19.661954 26472 init.cc:377] Logged in from keytab as kudu/127.3.183.254@KRBTEST.COM (short username kudu)
I20260929 17:52:19.662897 26472 webserver.cc:460] Webserver started at http://127.3.183.254:43595/ using document root <none> and password file <none>
I20260929 17:52:19.663422 26472 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:19.663480 26472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:19.663647 26472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:19.665820 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data/instance:
uuid: "808f62276c784b1682d7e535d09f99f6"
format_stamp: "Formatted at 2026-09-29 17:52:19 on dist-test-slave-wbpt"
server_key: "b20e43af0ec765acf4879b8e25cba293"
server_key_iv: "c641b1bfc0f9e9df478d3dcce9cc719f"
server_key_version: "encryptionkey@0"
I20260929 17:52:19.666390 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/wal/instance:
uuid: "808f62276c784b1682d7e535d09f99f6"
format_stamp: "Formatted at 2026-09-29 17:52:19 on dist-test-slave-wbpt"
server_key: "b20e43af0ec765acf4879b8e25cba293"
server_key_iv: "c641b1bfc0f9e9df478d3dcce9cc719f"
server_key_version: "encryptionkey@0"
I20260929 17:52:19.670428 26472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 17:52:19.672806 26487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:19.674033 26472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:19.674146 26472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/wal
uuid: "808f62276c784b1682d7e535d09f99f6"
format_stamp: "Formatted at 2026-09-29 17:52:19 on dist-test-slave-wbpt"
server_key: "b20e43af0ec765acf4879b8e25cba293"
server_key_iv: "c641b1bfc0f9e9df478d3dcce9cc719f"
server_key_version: "encryptionkey@0"
I20260929 17:52:19.674238 26472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:19.689124 26472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:19.692305 26472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:19.692487 26472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:19.699049 26472 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.254:36393
I20260929 17:52:19.699318 26539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.254:36393 every 8 connection(s)
I20260929 17:52:19.700070 26472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:19.702481 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 26472
I20260929 17:52:19.702607 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-0/wal/instance
I20260929 17:52:19.702888 3807 external_mini_cluster.cc:1428] Setting key 9824698524ed4f86deadb1a40fe188b9
I20260929 17:52:19.702888 26540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:19.707005 26540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.183.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.253@KRBTEST.COM" created.
Sep 29 17:52:19 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:19.722060 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.709373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.3.183.254:36393)
0929 17:52:19.709789 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:19.709808 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:19.710045 (+ 237us) server_negotiation.cc:407] Connection header received
0929 17:52:19.711104 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:19.711108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:19.711317 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:19.711494 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:19.712368 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.713312 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.714449 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.714878 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.715011 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.717544 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:19.717549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:19.717560 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:19.717581 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:19.719038 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.719459 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.719463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.719464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.719576 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.719864 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.719867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.719869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.720249 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:19.720400 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:19.720730 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 17:52:19.720956 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260929 17:52:19.722060 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.709382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36393 (local address 127.0.0.1:49242)
0929 17:52:19.709913 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.709948 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:19.710801 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:19.711508 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:19.711513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:19.711951 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:19.712225 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.712239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.713420 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.713423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.714302 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.714313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.715039 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.715043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.715158 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.715757 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:19.715798 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:19.717370 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:19.719136 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.719144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.719154 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.719362 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.719657 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.719660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.719662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.719766 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.720352 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:19.720359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:19.720500 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 17:52:19.720640 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 17:52:19.721053 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":101,"spinlock_wait_cycles":31104,"thread_start_us":169,"threads_started":1}
I20260929 17:52:19.724557 26540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:19.725286 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.725014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52142)
0929 17:52:19.725128 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.725206 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 17:52:19.725517 26491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab.
Entry for principal kudu/127.3.183.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab.
I20260929 17:52:19.726343 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.726096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52144)
0929 17:52:19.726221 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.726277 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 17:52:19.726500 26540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:45421: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
I20260929 17:52:19.727092 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:45421
--webserver_interface=127.3.183.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab
--principal=kudu/127.3.183.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:36393,127.3.183.253:45421,127.3.183.252:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:52:19.775842 26540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } attempt: 1
I20260929 17:52:19.776749 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.776461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52150)
0929 17:52:19.776610 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.776671 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 17:52:19.777644 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.777302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52152)
0929 17:52:19.777524 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.777580 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260929 17:52:19.777801 26540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:45421: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
W20260929 17:52:19.838188 26547 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:52:19.838502 26547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:19.838526 26547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:19.838562 26547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:19.842593 26547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:19.842659 26547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:19.842677 26547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:19.842692 26547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:19.842705 26547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:19.845988 26540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } attempt: 2
I20260929 17:52:19.846771 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.846481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52164)
0929 17:52:19.846634 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.846695 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 17:52:19.847582 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.847314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52176)
0929 17:52:19.847444 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.847517 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 17:52:19.847720 26540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.253:45421: Network error: Client connection negotiation failed: client connection to 127.3.183.253:45421: connect: Connection refused (error 111)
I20260929 17:52:19.848428 26547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:36393,127.3.183.253:45421,127.3.183.252:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.253.keytab
--principal=kudu/127.3.183.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.253:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.183.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:19.849447 26547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:19.850419 26547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:52:19.859050 26547 server_base.cc:1034] running on GCE node
W20260929 17:52:19.858999 26552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:19.859031 26553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:19.858999 26555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:19.859872 26547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:19.861008 26547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:19.862177 26547 hybrid_clock.cc:648] HybridClock initialized: now 1790704339862150 us; error 45 us; skew 500 ppm
Sep 29 17:52:19 dist-test-slave-wbpt krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:19.865295 26547 init.cc:377] Logged in from keytab as kudu/127.3.183.253@KRBTEST.COM (short username kudu)
I20260929 17:52:19.866184 26547 webserver.cc:460] Webserver started at http://127.3.183.253:43787/ using document root <none> and password file <none>
I20260929 17:52:19.866711 26547 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:19.866775 26547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:19.866974 26547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:19.869010 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data/instance:
uuid: "9208ec267561420d92d789418fa59cd8"
format_stamp: "Formatted at 2026-09-29 17:52:19 on dist-test-slave-wbpt"
server_key: "e39489ea6449ca194d1489ba5d6d86e7"
server_key_iv: "b01475a093dfce4fb3aa2caa2d391126"
server_key_version: "encryptionkey@0"
I20260929 17:52:19.869526 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/wal/instance:
uuid: "9208ec267561420d92d789418fa59cd8"
format_stamp: "Formatted at 2026-09-29 17:52:19 on dist-test-slave-wbpt"
server_key: "e39489ea6449ca194d1489ba5d6d86e7"
server_key_iv: "b01475a093dfce4fb3aa2caa2d391126"
server_key_version: "encryptionkey@0"
I20260929 17:52:19.873270 26547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:52:19.875687 26562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:19.876961 26547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:19.877120 26547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/wal
uuid: "9208ec267561420d92d789418fa59cd8"
format_stamp: "Formatted at 2026-09-29 17:52:19 on dist-test-slave-wbpt"
server_key: "e39489ea6449ca194d1489ba5d6d86e7"
server_key_iv: "b01475a093dfce4fb3aa2caa2d391126"
server_key_version: "encryptionkey@0"
I20260929 17:52:19.877223 26547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:19.907207 26547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:19.910410 26547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:19.910604 26547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:19.917683 26547 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.253:45421
I20260929 17:52:19.917943 26614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.253:45421 every 8 connection(s)
I20260929 17:52:19.918695 26547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/data/info.pb
I20260929 17:52:19.921561 26615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:19.923538 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 26547
I20260929 17:52:19.923671 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-1/wal/instance
I20260929 17:52:19.924003 3807 external_mini_cluster.cc:1428] Setting key c9bea3c04e63e033673ea3907747accd
I20260929 17:52:19.926241 26615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.183.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.183.252@KRBTEST.COM" created.
Sep 29 17:52:19 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:19.938889 26540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } attempt: 3
I20260929 17:52:19.940676 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.928216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.3.183.254:36393)
0929 17:52:19.928338 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:19.928341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:19.929406 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 17:52:19.930358 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:19.930362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:19.930423 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:19.930497 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:19.931693 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.932265 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.933417 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.933691 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.933778 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.936422 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:19.936425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:19.936427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:19.936466 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:19.938058 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.938648 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.938652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.938654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.938750 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.939163 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.939167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.939169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.939405 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:19.939488 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:19.940417 (+ 929us) server_negotiation.cc:299] Negotiation successful
0929 17:52:19.940554 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:52:19.942327 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.928742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36393 (local address 127.0.0.1:49258)
0929 17:52:19.929256 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.929298 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:19.930141 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:19.930671 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:19.930678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:19.931041 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:19.931555 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.931570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.932383 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.932386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.933294 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.933304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.933841 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.933844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.933999 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.934624 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:19.934640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:19.936201 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:19.938192 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.938198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.938214 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.938541 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.938845 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.938847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.938848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.939016 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.939730 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:19.939737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:19.940125 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 17:52:19.940884 (+ 759us) client_negotiation.cc:240] Negotiation successful
0929 17:52:19.941249 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":156,"threads_started":1}
I20260929 17:52:19.944540 26615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:52:19 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
Sep 29 17:52:19 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
I20260929 17:52:19.950623 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.939342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52192)
0929 17:52:19.939571 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.939582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:19.939634 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:19.940327 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:19.940331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:19.940534 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:19.940657 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.940665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.942139 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.942143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.942935 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.942948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.943729 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.943733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.943873 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.944404 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:19.944416 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:19.946325 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:19.948848 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.948851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.948854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.949078 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.949395 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.949398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.949399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.949494 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.950040 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:19.950043 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:19.950143 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 17:52:19.950257 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 17:52:19.950470 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 17:52:19.951334 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.939501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.3.183.253:45421)
0929 17:52:19.939877 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:19.939889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:19.939914 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:52:19.939953 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:19.939960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:19.940178 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:19.940328 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:19.940801 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.941997 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.943067 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.943580 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.943700 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.946508 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:19.946519 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:19.946535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:19.946557 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:19.948731 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.949206 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.949213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.949215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.949292 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.949570 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.949573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.949574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.949908 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:19.950048 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:19.950909 (+ 861us) server_negotiation.cc:299] Negotiation successful
0929 17:52:19.951110 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.183.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab.
Entry for principal kudu/127.3.183.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab.
I20260929 17:52:19.953289 3807 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6qiWXY/build/debug/bin/kudu
/tmp/dist-test-task6qiWXY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:38901
--webserver_interface=127.3.183.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab
--principal=kudu/127.3.183.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.183.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.183.254:36393,127.3.183.253:45421,127.3.183.252:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:52:19.953358 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.945092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52208)
0929 17:52:19.945220 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.945234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:19.945319 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:19.945735 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:19.945738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:19.945885 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:19.945985 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.945990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.946773 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.946776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.947307 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:19.947315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.947761 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.947764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:19.947880 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.948275 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:19.948286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:19.949771 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:19.951773 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.951776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.951778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.952039 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.952432 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:19.952435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:19.952437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:19.952530 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:19.952888 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:19.952891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:19.952971 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 17:52:19.953053 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 17:52:19.953197 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 17:52:19.953428 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.945161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.3.183.253:45421)
0929 17:52:19.945497 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:19.945506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:19.945521 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:52:19.945558 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:19.945561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:19.945603 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:19.945688 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:19.946095 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.946679 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.947412 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:19.947655 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:19.947751 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:19.949929 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:19.949932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:19.949935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:19.949956 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:19.951677 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.952179 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.952183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.952185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.952276 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:19.952616 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:19.952618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:19.952620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:19.952791 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:19.952874 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:19.953105 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 17:52:19.953258 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":260,"thread_start_us":146,"threads_started":1}
I20260929 17:52:19.953894 26540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:19.954336 26615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:19.954698 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.954397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60046)
0929 17:52:19.954570 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.954622 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 17:52:19.954998 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.954718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60052)
0929 17:52:19.954868 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.954936 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 17:52:19.955163 26566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
I20260929 17:52:19.955358 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.955124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60062)
0929 17:52:19.955259 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.955303 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 17:52:19.955489 26540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38901: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
I20260929 17:52:19.956120 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.955921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60078)
0929 17:52:19.956014 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.956061 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 17:52:19.956249 26615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38901: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
I20260929 17:52:19.986670 26540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } attempt: 1
I20260929 17:52:19.987516 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.987220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60082)
0929 17:52:19.987372 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.987434 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 17:52:19.988271 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:19.988059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60084)
0929 17:52:19.988163 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:19.988210 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 17:52:19.988420 26540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38901: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
I20260929 17:52:20.005553 26615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } attempt: 1
I20260929 17:52:20.006552 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.006190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60096)
0929 17:52:20.006374 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.006455 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 17:52:20.007371 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.007117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60112)
0929 17:52:20.007235 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.007301 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 17:52:20.007547 26615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38901: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
I20260929 17:52:20.063602 26540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } attempt: 2
I20260929 17:52:20.064512 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.064216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60122)
0929 17:52:20.064361 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.064423 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 17:52:20.065188 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.064979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60128)
0929 17:52:20.065080 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.065136 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 17:52:20.065343 26540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38901: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
W20260929 17:52:20.068755 26622 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:52:20.069060 26622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:20.069104 26622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:20.069167 26622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:20.073395 26622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:20.073483 26622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:20.073513 26622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:20.073542 26622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:20.073571 26622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:20.075744 26615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } attempt: 2
I20260929 17:52:20.076568 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.076247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60140)
0929 17:52:20.076408 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.076479 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 17:52:20.077227 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.077016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60144)
0929 17:52:20.077119 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.077167 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 17:52:20.077335 26615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.183.252:38901: Network error: Client connection negotiation failed: client connection to 127.3.183.252:38901: connect: Connection refused (error 111)
I20260929 17:52:20.079406 26622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.183.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.183.254:36393,127.3.183.253:45421,127.3.183.252:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/krb5kdc/kudu_127.3.183.252.keytab
--principal=kudu/127.3.183.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.183.252:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.183.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:20.080515 26622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:20.081528 26622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:20.089835 26628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:20.089872 26622 server_base.cc:1034] running on GCE node
W20260929 17:52:20.089831 26631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:20.089831 26629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:20.090639 26622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:20.091761 26622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:20.092979 26622 hybrid_clock.cc:648] HybridClock initialized: now 1790704340092951 us; error 39 us; skew 500 ppm
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:20.095834 26622 init.cc:377] Logged in from keytab as kudu/127.3.183.252@KRBTEST.COM (short username kudu)
I20260929 17:52:20.096698 26622 webserver.cc:460] Webserver started at http://127.3.183.252:45887/ using document root <none> and password file <none>
I20260929 17:52:20.097231 26622 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:20.097292 26622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:20.097472 26622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:20.099496 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data/instance:
uuid: "3ca18540184c40968ecdab8d67d4284d"
format_stamp: "Formatted at 2026-09-29 17:52:20 on dist-test-slave-wbpt"
server_key: "b4d2b6e448633efcd11d7c504f5ee0f4"
server_key_iv: "0dcb793c5dcd91f618096f8eac1dca81"
server_key_version: "encryptionkey@0"
I20260929 17:52:20.100082 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/wal/instance:
uuid: "3ca18540184c40968ecdab8d67d4284d"
format_stamp: "Formatted at 2026-09-29 17:52:20 on dist-test-slave-wbpt"
server_key: "b4d2b6e448633efcd11d7c504f5ee0f4"
server_key_iv: "0dcb793c5dcd91f618096f8eac1dca81"
server_key_version: "encryptionkey@0"
I20260929 17:52:20.103950 26622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:52:20.106492 26638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:20.107740 26622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:52:20.107892 26622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data,/tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/wal
uuid: "3ca18540184c40968ecdab8d67d4284d"
format_stamp: "Formatted at 2026-09-29 17:52:20 on dist-test-slave-wbpt"
server_key: "b4d2b6e448633efcd11d7c504f5ee0f4"
server_key_iv: "0dcb793c5dcd91f618096f8eac1dca81"
server_key_version: "encryptionkey@0"
I20260929 17:52:20.108011 26622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:20.123021 26622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:20.126156 26622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:20.126332 26622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:20.133221 26622 rpc_server.cc:307] RPC server started. Bound to: 127.3.183.252:38901
I20260929 17:52:20.133581 26690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.183.252:38901 every 8 connection(s)
I20260929 17:52:20.134228 26622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/data/info.pb
I20260929 17:52:20.137065 26691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:20.140193 3807 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6qiWXY/build/debug/bin/kudu as pid 26622
I20260929 17:52:20.140321 3807 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6qiWXY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704058755242-3807-0/minicluster-data/master-2/wal/instance
I20260929 17:52:20.140573 3807 external_mini_cluster.cc:1428] Setting key 9ef89cce624914d6fb37567a6574cade
I20260929 17:52:20.141343 26691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:52:20.153548 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.143389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.3.183.254:36393)
0929 17:52:20.143552 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.143556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.143935 (+ 379us) server_negotiation.cc:407] Connection header received
0929 17:52:20.144133 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.144136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.144191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.144266 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.145183 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.145692 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.147118 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.147343 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.147425 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.149989 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.149992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.149994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.150020 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.151484 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.152077 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.152079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.152080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.152150 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.152473 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.152476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.152478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.152672 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.152754 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.153261 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.153379 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 17:52:20.154465 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.142053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.3.183.252:38901)
0929 17:52:20.142532 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.142548 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.142586 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:52:20.143429 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.143453 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.143953 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.144234 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.145012 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.146039 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.146975 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.147507 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.147697 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.150354 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.150361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.150376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.150400 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.152112 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.152509 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.152514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.152516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.152588 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.152819 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.152822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.152824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.153076 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.153254 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.153458 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.153591 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":332,"thread_start_us":143,"threads_started":1}
I20260929 17:52:20.154470 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.143337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.254:36393 (local address 127.0.0.1:49262)
0929 17:52:20.143768 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.143844 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:20.143988 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:20.144338 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:20.144343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:20.144748 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:20.145039 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.145052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.145795 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.145799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.146975 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.146984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.147494 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.147499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.147701 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.148351 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:20.148368 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:20.149834 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:20.151598 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.151606 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.151618 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.151990 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.152237 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.152240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.152242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.152378 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.152767 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:20.152774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:20.153028 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 17:52:20.153170 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 17:52:20.153507 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":39,"spinlock_wait_cycles":7424,"thread_start_us":168,"threads_started":1}
I20260929 17:52:20.155738 26691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.254@KRBTEST.COM
I20260929 17:52:20.164446 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.156417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.3.183.253:45421)
0929 17:52:20.156546 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.156552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.156565 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:52:20.156609 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.156612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.156662 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.156738 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.157174 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.157754 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.158492 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.158732 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.158822 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.161174 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.161177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.161179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.161204 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.162801 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.163250 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.163253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.163254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.163326 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.163608 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.163611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.163613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.163844 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.163937 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.164165 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.164290 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:52:20.164477 26540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } attempt: 3
I20260929 17:52:20.164422 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.156263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.253:45421 (local address 127.0.0.1:52214)
0929 17:52:20.156419 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.156432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:20.156507 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:20.156762 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:20.156765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:20.156945 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:20.157077 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.157083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.157840 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.157843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.158372 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.158379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.158829 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.158833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.158924 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.159345 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:20.159359 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:20.160623 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:20.162929 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.162932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.162934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.163148 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.163422 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.163425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.163427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.163526 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.163925 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:20.163928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:20.164013 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 17:52:20.164092 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 17:52:20.164238 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 17:52:20.165185 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.156828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.3.183.254:36393)
0929 17:52:20.156945 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.156949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.156978 (+ 29us) server_negotiation.cc:407] Connection header received
0929 17:52:20.157105 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.157108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.157151 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.157218 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.157753 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.158267 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.158998 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.159208 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.159288 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.161578 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.161582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.161585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.161611 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.163075 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.163501 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.163503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.163504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.163573 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.163835 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.163838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.163840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.164111 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.164356 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.164461 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.165050 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:52:20.165365 26691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:52:20.168471 26615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } attempt: 3
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.254@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.252@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.183.253@KRBTEST.COM
I20260929 17:52:20.174685 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.164855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60174)
0929 17:52:20.164978 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.164992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:20.165064 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:20.165410 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:20.165413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:20.165569 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:20.165664 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.165671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.166600 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.166603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.167383 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.167393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.167872 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.167876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.167957 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.168372 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:20.168383 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:20.170069 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:20.172138 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.172141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.172143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.172379 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.173379 (+ 1000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.173382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.173384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.173487 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.174205 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:20.174208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:20.174304 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 17:52:20.174382 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 17:52:20.174527 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
Sep 29 17:52:20 dist-test-slave-wbpt krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.183.253@KRBTEST.COM for kudu/127.3.183.252@KRBTEST.COM
I20260929 17:52:20.175154 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.165797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60178)
0929 17:52:20.165892 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.165904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:20.165964 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:20.166457 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:20.166459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:20.166622 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:20.166701 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.166707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.168384 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.168388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.169146 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.169155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.169764 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.169767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.169839 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.170191 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:20.170203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:20.172060 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:20.173790 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.173792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.173795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.174008 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.174253 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.174255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.174257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.174322 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.174713 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:20.174715 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:20.174790 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 17:52:20.174876 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 17:52:20.175000 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 17:52:20.175295 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.165080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.3.183.252:38901)
0929 17:52:20.165175 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.165179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.165190 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:52:20.165221 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.165224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.165267 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.165386 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.165825 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.166520 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.167483 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.167727 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.167841 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.170312 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.170316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.170318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.170344 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.172009 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.172836 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.172839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.172841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.173263 (+ 422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.173601 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.173603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.173605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.174095 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.174952 (+ 857us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.175021 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.175127 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":11136}
I20260929 17:52:20.175644 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.165933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.3.183.252:38901)
0929 17:52:20.166209 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.166217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.166228 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:52:20.166258 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.166260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.166296 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.166370 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.166780 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.168034 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.169229 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.169585 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.169672 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.172171 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.172174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.172176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.172196 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.173700 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.174102 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.174105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.174107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.174175 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.174406 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.174409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.174410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.174595 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.174681 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.175373 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.175477 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":130,"threads_started":1}
I20260929 17:52:20.176409 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.166285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.3.183.253:45421)
0929 17:52:20.166383 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.166386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.166454 (+ 68us) server_negotiation.cc:407] Connection header received
0929 17:52:20.166563 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.166566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.166608 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.166676 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.167364 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.168095 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.169152 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.169420 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.169473 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.172309 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.172314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.172317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.172342 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.173890 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.174334 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.174338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.174339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.174416 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.174693 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.174696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.174698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.174887 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.175071 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.176148 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.176261 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:52:20.178570 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.168801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.183.252:38901 (local address 127.0.0.1:60180)
0929 17:52:20.169676 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:20.169688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:20.169749 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:20.170563 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:20.170566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:20.170724 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:20.170815 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.170822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.171966 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.171969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.172730 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:20.172742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.173456 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.173461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:20.173564 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.173953 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:20.173968 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:20.175223 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:20.177096 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.177099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.177101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.177307 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.177639 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:20.177642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:20.177644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:20.177741 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:20.178141 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:20.178145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:20.178227 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 17:52:20.178300 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 17:52:20.178414 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804}
I20260929 17:52:20.178722 26540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6: Bootstrap starting.
I20260929 17:52:20.178848 26691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: Bootstrap starting.
I20260929 17:52:20.179067 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.169006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.3.183.252:38901)
0929 17:52:20.170333 (+ 1327us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.170342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.170354 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:52:20.170383 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.170386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.170421 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.170921 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.171033 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.171851 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.173003 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.173331 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.173744 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.175359 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.175362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.175365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.175383 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.176955 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.177466 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.177469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.177471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.177544 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.177845 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.177848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.177849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.178037 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.178752 (+ 715us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.178821 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.178918 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":97,"threads_started":1}
I20260929 17:52:20.181491 26540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:20.182266 3807 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:52:20.182302 26540 log.cc:824] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:20.182448 26615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: Bootstrap starting.
I20260929 17:52:20.184417 26540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6: No bootstrap required, opened a new log
I20260929 17:52:20.184572 26615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:20.183367 26514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "808f62276c784b1682d7e535d09f99f6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49266
I20260929 17:52:20.185217 26615 log.cc:824] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:20.186368 26691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:20.186936 26540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.187104 26540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808f62276c784b1682d7e535d09f99f6, State: Initialized, Role: FOLLOWER
I20260929 17:52:20.187161 26691 log.cc:824] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:20.187384 26540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.188062 26615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: No bootstrap required, opened a new log
I20260929 17:52:20.188652 26540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:20.188745 26514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 17:52:20.188843 26514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:20.189252 26691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: No bootstrap required, opened a new log
I20260929 17:52:20.189965 26514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.192055 26615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.192111 26514 leader_election.cc:288] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [CANDIDATE]: Term 1 election: Requested vote from peers 9208ec267561420d92d789418fa59cd8 (127.3.183.253:45421), 3ca18540184c40968ecdab8d67d4284d (127.3.183.252:38901)
I20260929 17:52:20.192306 26615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9208ec267561420d92d789418fa59cd8, State: Initialized, Role: FOLLOWER
I20260929 17:52:20.192675 26615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.193573 26691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.193794 26691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca18540184c40968ecdab8d67d4284d, State: Initialized, Role: FOLLOWER
W20260929 17:52:20.194012 26493 tablet.cc:2369] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:52:20.194090 26691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.194651 26615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:20.194716 26589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "808f62276c784b1682d7e535d09f99f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9208ec267561420d92d789418fa59cd8"
I20260929 17:52:20.194952 26589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:20.195268 26691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:20.195859 26665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "808f62276c784b1682d7e535d09f99f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ca18540184c40968ecdab8d67d4284d"
I20260929 17:52:20.196082 26665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:20.193205 26472 master_runner.cc:186] Error getting master registration for 127.3.183.253:45421: OK, instance_id { permanent_uuid: "9208ec267561420d92d789418fa59cd8" instance_seqno: 1790704339915682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 17:52:20.196766 26472 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:52:20.197376 26665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808f62276c784b1682d7e535d09f99f6 in term 1.
I20260929 17:52:20.197827 26491 leader_election.cc:302] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca18540184c40968ecdab8d67d4284d, 808f62276c784b1682d7e535d09f99f6; no voters:
I20260929 17:52:20.198191 26589 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808f62276c784b1682d7e535d09f99f6 in term 1.
I20260929 17:52:20.199088 26713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.199214 26713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 17:52:20.200106 26712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:20.200170 26712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 17:52:20.201882 26742 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:20.201967 26742 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 17:52:20.202971 26739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:20.203049 26739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:52:20.203075 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.203204 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:20.203662 26701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.203797 26701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:20.203896 26701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:20.203975 26701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [term 1 LEADER]: Becoming Leader. State: Replica: 808f62276c784b1682d7e535d09f99f6, State: Running, Role: LEADER
I20260929 17:52:20.204279 26701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } }
I20260929 17:52:20.205152 26622 master_runner.cc:263] No leader master found from master 3ca18540184c40968ecdab8d67d4284d
I20260929 17:52:20.205217 26622 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:52:20.206640 26547 master_runner.cc:263] No leader master found from master 9208ec267561420d92d789418fa59cd8
I20260929 17:52:20.206697 26547 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:52:20.208426 26701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808f62276c784b1682d7e535d09f99f6. Latest consensus state: current_term: 1 leader_uuid: "808f62276c784b1682d7e535d09f99f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.208559 26701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:20.213330 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.202620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.3.183.252:38901)
0929 17:52:20.202716 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.202720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.205116 (+ 2396us) server_negotiation.cc:407] Connection header received
0929 17:52:20.205284 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.205287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.205334 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.205381 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.206298 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.206993 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.208114 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.208444 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.208519 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.210259 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.210263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.210266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.210291 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.211655 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.212120 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.212124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.212126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.212203 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.212490 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.212493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.212495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.212692 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.212818 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.213060 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.213172 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:52:20.216012 26749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:20.216995 26749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:20.218182 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.200566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.3.183.254:36393)
0929 17:52:20.200669 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.200673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.203318 (+ 2645us) server_negotiation.cc:407] Connection header received
0929 17:52:20.203513 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.203516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.203563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.203611 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.204825 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.205571 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.212870 (+ 7299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.213213 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.213291 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.214866 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.214870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.214873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.214898 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.216324 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.216774 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.216777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.216779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.216864 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.217186 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.217190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.217192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.217411 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.217542 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.217914 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.218032 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:52:20.220041 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:20.201542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.3.183.253:45421)
0929 17:52:20.201722 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.201726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.204757 (+ 3031us) server_negotiation.cc:407] Connection header received
0929 17:52:20.206849 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.206852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.206897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.206944 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.211877 (+ 4933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.212645 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.214824 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:20.215170 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:20.215243 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:20.216763 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:20.216767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:20.216770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:20.216796 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:20.218325 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.218739 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.218743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.218744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.218836 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:20.219125 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:20.219128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:20.219130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:20.219336 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:20.219470 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:20.219754 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 17:52:20.219887 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 17:52:20.233635 26589 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [term 1 FOLLOWER]: Refusing update from remote peer 808f62276c784b1682d7e535d09f99f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:52:20.233671 26665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [term 1 FOLLOWER]: Refusing update from remote peer 808f62276c784b1682d7e535d09f99f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:52:20.234371 26701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:52:20.234568 26740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:52:20.238515 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808f62276c784b1682d7e535d09f99f6. Latest consensus state: current_term: 1 leader_uuid: "808f62276c784b1682d7e535d09f99f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.238667 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:20.239085 26713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808f62276c784b1682d7e535d09f99f6. Latest consensus state: current_term: 1 leader_uuid: "808f62276c784b1682d7e535d09f99f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.239207 26713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:20.250720 26701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "808f62276c784b1682d7e535d09f99f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.250878 26701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:20.251929 26750 mvcc.cc:205] Tried to move back new op lower bound from 7334724977587879936 to 7334724977489567744. Current Snapshot: MvccSnapshot[applied={T|T < 7334724977587879936}]
I20260929 17:52:20.252199 26740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "808f62276c784b1682d7e535d09f99f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.252321 26740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:20.253389 26713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "808f62276c784b1682d7e535d09f99f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.253530 26713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:20.254370 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "808f62276c784b1682d7e535d09f99f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "808f62276c784b1682d7e535d09f99f6" member_type: VOTER last_known_addr { host: "127.3.183.254" port: 36393 } } peers { permanent_uuid: "9208ec267561420d92d789418fa59cd8" member_type: VOTER last_known_addr { host: "127.3.183.253" port: 45421 } } peers { permanent_uuid: "3ca18540184c40968ecdab8d67d4284d" member_type: VOTER last_known_addr { host: "127.3.183.252" port: 38901 } } }
I20260929 17:52:20.254503 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:52:20.254891 26752 mvcc.cc:205] Tried to move back new op lower bound from 7334724977587879936 to 7334724977489567744. Current Snapshot: MvccSnapshot[applied={T|T < 7334724977587879936}]
I20260929 17:52:20.263866 26749 catalog_manager.cc:1347] Generated new cluster ID: 273b82aba47d4337b5b00f591821f191
I20260929 17:52:20.263967 26749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:20.281258 26749 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:20.282601 26749 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:20.300550 26749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 808f62276c784b1682d7e535d09f99f6: Generated new TSK 0
I20260929 17:52:20.301544 26749 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:52:20.337419 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.334787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.3.183.253:45421)
0929 17:52:20.334912 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.334917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.336107 (+ 1190us) server_negotiation.cc:407] Connection header received
0929 17:52:20.336559 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.336562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.336609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.336657 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.337288 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52236: BlockingRecv error: recv got EOF from 127.0.0.1:52236 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:52:20.338315 26697 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.335481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.3.183.252:38901)
0929 17:52:20.335586 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.335590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.336413 (+ 823us) server_negotiation.cc:407] Connection header received
0929 17:52:20.336518 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.336521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.336571 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.336640 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.338194 (+ 1554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60196: BlockingRecv error: recv got EOF from 127.0.0.1:60196 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 17:52:20.340449 26543 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.339172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.3.183.254:36393)
0929 17:52:20.339281 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.339285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.339568 (+ 283us) server_negotiation.cc:407] Connection header received
0929 17:52:20.339726 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.339730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.339799 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.339848 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.340329 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49282: BlockingRecv error: recv got EOF from 127.0.0.1:49282 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 17:52:20.344799 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.343391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.3.183.253:45421)
0929 17:52:20.343491 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.343495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.343897 (+ 402us) server_negotiation.cc:407] Connection header received
0929 17:52:20.344113 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.344116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.344190 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.344238 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.344696 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52248: BlockingRecv error: recv got EOF from 127.0.0.1:52248 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 17:52:20.349860 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.348390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.3.183.253:45421)
0929 17:52:20.348492 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.348496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.348991 (+ 495us) server_negotiation.cc:407] Connection header received
0929 17:52:20.349177 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.349181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.349231 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.349281 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.349750 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52250: BlockingRecv error: recv got EOF from 127.0.0.1:52250 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 17:52:20.355365 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.353858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.3.183.253:45421)
0929 17:52:20.353968 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.353972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.354460 (+ 488us) server_negotiation.cc:407] Connection header received
0929 17:52:20.354641 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.354645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.354696 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.354748 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.355266 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52266: BlockingRecv error: recv got EOF from 127.0.0.1:52266 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 17:52:20.363171 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.361534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.3.183.253:45421)
0929 17:52:20.361689 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.361695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.362249 (+ 554us) server_negotiation.cc:407] Connection header received
0929 17:52:20.362436 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.362440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.362494 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.362546 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.363065 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52276: BlockingRecv error: recv got EOF from 127.0.0.1:52276 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 17:52:20.375535 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.373592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.3.183.253:45421)
0929 17:52:20.373713 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.373718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.374342 (+ 624us) server_negotiation.cc:407] Connection header received
0929 17:52:20.374558 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.374562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.374619 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.374672 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.375419 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52282: BlockingRecv error: recv got EOF from 127.0.0.1:52282 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 17:52:20.396003 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.394201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.3.183.253:45421)
0929 17:52:20.394326 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.394331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.394981 (+ 650us) server_negotiation.cc:407] Connection header received
0929 17:52:20.395190 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.395194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.395249 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.395302 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.395875 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52286: BlockingRecv error: recv got EOF from 127.0.0.1:52286 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 17:52:20.431979 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.430603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.3.183.253:45421)
0929 17:52:20.430777 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.430793 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.430942 (+ 149us) server_negotiation.cc:407] Connection header received
0929 17:52:20.431106 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.431109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.431160 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.431234 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.431881 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52294: BlockingRecv error: recv got EOF from 127.0.0.1:52294 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 17:52:20.499130 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.498091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.3.183.253:45421)
0929 17:52:20.498234 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.498239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.498362 (+ 123us) server_negotiation.cc:407] Connection header received
0929 17:52:20.498479 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.498482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.498536 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.498612 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.499010 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52302: BlockingRecv error: recv got EOF from 127.0.0.1:52302 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 17:52:20.630319 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.629165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.3.183.253:45421)
0929 17:52:20.629315 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.629320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.629501 (+ 181us) server_negotiation.cc:407] Connection header received
0929 17:52:20.629649 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.629652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.629699 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.629769 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.630222 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52318: BlockingRecv error: recv got EOF from 127.0.0.1:52318 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 17:52:20.889846 26621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:52:20.888745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.3.183.253:45421)
0929 17:52:20.888881 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:20.888884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:20.889029 (+ 145us) server_negotiation.cc:407] Connection header received
0929 17:52:20.889168 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:20.889171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:20.889220 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:20.889320 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:20.889724 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52326: BlockingRecv error: recv got EOF from 127.0.0.1:52326 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:52:21.203210 26742 catalog_manager.cc:1259] Loaded cluster ID: 273b82aba47d4337b5b00f591821f191
I20260929 17:52:21.203301 26742 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: loading cluster ID for follower catalog manager: success
I20260929 17:52:21.204201 26739 catalog_manager.cc:1259] Loaded cluster ID: 273b82aba47d4337b5b00f591821f191
I20260929 17:52:21.204262 26739 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: loading cluster ID for follower catalog manager: success
I20260929 17:52:21.205076 26742 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: acquiring CA information for follower catalog manager: success
I20260929 17:52:21.205825 26739 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: acquiring CA information for follower catalog manager: success
I20260929 17:52:21.206270 26742 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9208ec267561420d92d789418fa59cd8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:52:21.206872 26739 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ca18540184c40968ecdab8d67d4284d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:52:21.408062 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:21.403979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.3.183.253:45421)
0929 17:52:21.404203 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:21.404208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:21.404375 (+ 167us) server_negotiation.cc:407] Connection header received
0929 17:52:21.404531 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:21.404534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:21.404662 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:21.404762 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:21.405092 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.405618 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.406528 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.406749 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.406837 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:21.407168 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:21.407607 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:21.407712 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:21.407813 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 17:52:21.407875 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260929 17:52:21.414978 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:21.411201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.3.183.252:38901)
0929 17:52:21.411450 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:21.411455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:21.411468 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:52:21.411643 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:21.411645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:21.411764 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:21.411927 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:21.412148 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.412741 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.413613 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.413837 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.413926 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:21.414187 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:21.414630 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:21.414724 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:21.414767 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 17:52:21.414817 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
I20260929 17:52:21.421526 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:21.417914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.3.183.253:45421)
0929 17:52:21.418031 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:21.418035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:21.418344 (+ 309us) server_negotiation.cc:407] Connection header received
0929 17:52:21.418403 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:21.418406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:21.418449 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:21.418540 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:21.418774 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.419445 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.420342 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.420684 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.420789 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:21.421069 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:21.421150 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:21.421222 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:21.421320 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 17:52:21.421388 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:52:21.422027 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:21.418276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.3.183.252:38901)
0929 17:52:21.418429 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:21.418434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:21.418790 (+ 356us) server_negotiation.cc:407] Connection header received
0929 17:52:21.418945 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:21.418948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:21.418998 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:21.419064 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:21.419287 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.420010 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.420955 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.421220 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.421292 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:21.421616 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:21.421692 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:21.421756 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:21.421853 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 17:52:21.421904 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 17:52:21.422490 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:21.417912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.3.183.254:36393)
0929 17:52:21.418176 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:21.418180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:21.418360 (+ 180us) server_negotiation.cc:407] Connection header received
0929 17:52:21.418464 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:21.418467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:21.418634 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:21.418743 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:52:21.418989 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.419755 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.420890 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:21.421158 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:21.421248 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:21.421739 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:52:21.422121 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:52:21.422203 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:21.422283 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 17:52:21.422343 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260929 17:52:21.429097 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 26472
I20260929 17:52:21.435325 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 26547
I20260929 17:52:21.440876 3807 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6qiWXY/build/debug/bin/kudu with pid 26622
2026-09-29T17:52:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2999 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6001 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282665 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST